Advanced lensless light-field imaging systems and methods for enabling a wide range of entirely new applications

ABSTRACT

Continuing a sequence of lensless light-field imaging camera patents beginning 1999, the present invention adds light-use efficiency, predictive-model design, distance-parameterized interpolation, computational efficiency, arbitrary shaped surface-of-focus, angular diversity/redundancy, distributed image sensing, plasmon surface propagation, and other fundamentally enabling features. Embodiments can be fabricated entirely by printing, transparent/semi-transparent, layered, of arbitrary size/curvature, flexible/bendable, emit light, focus and self-illuminate at zero-separation distance between (planar or curved) sensing and observed surfaces, robust against damage/occulation, implement color sensing without use of filters or diffraction, overlay on provided surfaces, provided color and enhanced multi-wavelength color sensing, wavelength-selective imaging of near-infrared/near-ultraviolet, and comprise many other fundamentally enabling features. Embodiments can be thinner, larger/smaller, more light-use efficient, and higher-performance than recently-popularized coded aperture imaging cameras. Vast ranges of diverse previously-impossible applications are enabled: credit-card cameras/phones, in-body monitoring of healing/disease, advanced biomarker analysis systems, perfect eye-contact video conferencing, seeing fabrics/skin/housings, and manufacturing-monitoring, wear-monitoring, and machine vision capabilities.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.62/360,472, filed Jul. 11, 2016, and U.S. Provisional Application No.62/528,384, filed Jul. 3, 2017, the disclosures of which areincorporated herein in their entireties by reference.

COPYRIGHT & TRADEMARK NOTICES

A portion of the disclosure of this patent document may containmaterial, which is subject to copyright protection. Certain marksreferenced herein may be common law or registered trademarks of theapplicant, the assignee or third parties affiliated or unaffiliated withthe applicant or the assignee. Use of these marks is for providing anenabling disclosure by way of example and shall not be construed toexclusively limit the scope of the disclosed subject matter to materialassociated with such marks.

BACKGROUND OF THE INVENTION Field of the Invention

The invention pertains to computational imaging and light-field sensors,and more specifically to lensless camera arrangements leveraging a widerange of curved, polygon, rigid, flexible, elastic, and plasticattributes.

Overview of the Invention

FIG. 1 depicts an example conceptual view of the underlying principlesof the invention, facilitating a wide range of implementation methodsand architectures. In this depiction, an Optical Scene creates aLight-Field that is directed to an Optical Sensor which is preceded byone or more Lensless Optical Structure(s) that in some manner alters thelight field in a predictable spatial manner. The Optical Sensor produces(typically time-varying) electrical signals and/or computational dataresponsive (instantly and/or within some time-delay) to light incidentto the surface or other substructure(s) within the Optical Sensor at anygiven moment. The depicted Inverse Model can be configured to, in someappropriate manner, undo the effects of the incoming light's opticaltravel first within the Light-Field preceding the optical structure(s)and then through the Lensless Optical Structure(s) to where it reachesthe Optical Sensor, resulting in a computationally-produced image which,for example, can be arranged to be useful for human or machine use.

A family of technologies relating to lensless imaging wherein an (evenprimitive) light sensing array is configured by (simple or more complex)optical structures to create a light-field sensor and focused images areobtained via numerical computation employing algorithms executed on oneor more instances of a computational environment (for example comprisinga computer, microprocessor, Graphical Processing Unit (GPU) chip,Digital Signal Processing (DSP) chip, etc.) has been described inearlier patent filings by the present inventor. These include forexample:

-   -   U.S. Pat. No. 9,172,850 “Lensless imaging camera performing        image formation in software employing micro-optic elements        creating overlap of light from distant sources over multiple        photo sensor elements”    -   U.S. Pat. No. 9,160,894 “Lensless imaging camera performing        image formation in software and employing micro-optic elements        that impose light diffractions”    -   U.S. Pat. No. 8,830,375 “Vignetted optoelectronic array for use        in synthetic image formation via signal processing, lensless        cameras, and integrated camera-displays”    -   U.S. Pat. No. 8,816,263 “Vignetted planar spatial light-field        sensor and spatial sampling designs for far-field lensless        synthetic imaging via signal processing image formation”    -   U.S. Pat. No. 8,754,842 “Combined display and image capture        without simple or compound lenses for video conferencing        eye-contact and other applications”    -   U.S. Pat. No. 8,305,480 “Synthetic Image Formation via Signal        Processing for Vignetted Optoelectronic Arrays, Lensless        Cameras, and Integrated Camera-Displays”    -   U.S. Pat. No. 8,284,290 “Synthetic Image Formation Signal        Processing Hardware for Vignetted Optoelectronic Arrays,        Lensless Cameras, and Integrated Camera-Displays”    -   U.S. Pat. No. 8,125,559 “Image Formation for Large Photosensor        Array Surfaces”    -   U.S. Pat. No. 9,019,237 “Multitouch Parameter And Gesture User        Interface Employing an LED-Array Tactile Sensor That Can Also        Operate as a Display”    -   C.A. 2,318,395 “Multifunction Communication Service Device”    -   U.S. Pat. No. 9,632,344 “Use of LED or OLED Array to Implement        Integrated Combinations of Touch Screen Tactile, Touch Gesture        Sensor, Color Image Display, Hand-Image Gesture Sensor, Document        Scanner, Secure Optical Data Exchange, and Fingerprint        Processing Capabilities”    -   U.S. application Ser. No. 13/547,024 “Use of OLED Displays as a        High-Resolution Optical Tactile Sensor for High Dimensional        Touchpad (HTTP) User Interfaces”    -   Allowed U.S. patent application Ser. No. 13/072,588 “Color        Imaging Using Color LED Array as Light-Field Image Sensor”    -   U.S. application Ser. No. 14/333,177 “Vignetted planar spatial        light-field sensor and spatial sampling designs for far-field        lensless synthetic imaging via signal processing image        formation”    -   U.S. application Ser. No. 14/478,920 “Vignetted Optoelectronic        Array for Use in Synthetic Image Formation via Signal        Processing, Lensless Cameras, and Integrated Camera-Displays”        as well as other past, current, and planned future patent        filings. The approach can also be used to implement lensless        microscopy and optical tomography, for example as has been        described in earlier patent filings by the inventor, for example        including:

U.S. Pat. No. 8,885,035 “Electronic imaging flow-microscope forenvironmental remote sensing, bioreactor process monitoring, and opticalmicroscopic tomography”

-   -   U.S. application Ser. No. 14/105,108 “Small-Profile Lensless        Optical Microscopy Imaging and Tomography Instruments and        Elements For Low Cost And Integrated Microscopy”    -   U.S. Pat. No. 9,594,239 “Optical Tomography for Microscopy, Cell        Cytometry, Microplate Array Instrumentation, Crystallography,        and Other Applications”    -   U.S. Pat. No. 9,594,019 “Optical Tomography for Microscopy, Cell        Cytometry, Microplate Array Instrumentation, Crystallography,        and Other Applications”    -   U.S. application Ser. No. 15/289,815 “Electronic Imaging        Flow-Microscope for Environmental Remote Sensing, Bioreactor        Process Monitoring, and Optical Microscopic Tomography”    -   U.S. application Ser. No. 15/457,963 “Cylindrical Optical        Tomography for Microscopy, Cell Cytometry, Microplate Array        Instrumentation, Crystallography, and Other Applications”        as well as other past, current, and planned future patent        filings. Broader uses and additional functions are possible as        taught in yet other past and current patent filings as well as        other planned and emergent patent filings.

As indicated in the above patent filings, an immense number ofcapabilities and features result from this approach. For example,although, the light sensing array can comprise a CMOS imaging chip, thelight sensing array can comprise an array of printedorganic-semiconductor photodiodes, including printed Organic LightEmitting Diodes (OLEDs) that are electrically interfaced and/orphysically structured to operate at least as light sensing (square)“pixel” or (rectangular) “pel” elements; because such a sensing array isfabricated by layered printing of inks (comprising conductive materials,semiconducting materials, and insulating materials, the types inksincluding transparent types) on arbitrary surfaces (such as glass, rigidplastics, or flexible plastics) that may be flat or curved, the opticalsensing array can be rendered in a wide variety of ways, shapes, sizes,curvatures, etc. on a flat, curved, bendable, deformable surface thatmay also be used for performing other function. Further, the opticalstructures required to invoke light-field sensing capabilities can be assimple as a crudely formed array of vignetting passages, and theseoptical structures can be for example printed using light-path impedinginks, an applicable light-field sensor array can be entirely fabricatedby printing layers of electrical, structural, and optical inks on flat,curved, bendable, and/or deformable surface that can also be used forperforming other structural, electrical, sensing, physically-supporting,physical-boundary, and/or physical-surface functions. It is noted thatin addition to printed organic-semiconductor photodiodes and/or OLEDs,the light sensing array can alternatively or additionally comprise oneor more of printed organic-semiconductor phototransistors, silicon orother crystal-lattice photodiodes, silicon or other crystal-latticephototransistors, silicon or other crystal-lattice LEDs, silicon orother crystal-lattice CMOS light sensors, charge-coupled light sensors,printed non-organic semiconductor photodiodes printed non-organicsemiconductor LEDs, printed non-organic semiconductor phototransistors,or other type of electrically-responsive light-sensing elements. Asdescribed and implied in the above patent materials, and as to befurther described and implied throughout the present patentapplications, these advanced lensless light-field imaging systems andmethods for enabling a wide range of entirely new applications.

These earlier inventor's patent families and the inventor's presentpatent application individually and collectively (1) employ manyindependent advancements in material science, organic electronics, andmanufacturing processes together with (2) novel adaptations of andstructures for optoelectronic devices, novel physical, optical,electronic and optoelectronic device configurations and (3)corresponding novel mathematical and signal flow structures arranged tobe implemented by signal processing algorithms, and other novel systemelements and method steps. The aforementioned independent advancementsin material science organic electronics, and manufacturing processesinclude:

-   -   Materials, methods, and manufacturing techniques for bendable        and flexible active and passive electronic and optoelectronic        components and interconnections;    -   Materials, methods, and manufacturing techniques for        printable/printed active and passive electronic and        optoelectronic components and interconnections.    -   Materials, methods, and manufacturing techniques for transparent        active and passive electronic and optoelectronic components and        interconnections;    -   Materials, methods, and manufacturing techniques for        multiply-layered/stackable active and passive electronic and        optoelectronic components and interconnections;    -   Materials, methods, structures, and manufacturing techniques for        implementing and optimizing light-sensing and light-emitting        aspects of optoelectronic components.

Novel adaptations of and structures for optoelectronic devices, novelphysical, optical, electronic and optoelectronic device configurationscomprised by the inventor's earlier patent families and used in theinventor's present patent application include for example but are notlimited to:

-   -   Use of diffraction elements, vignetting structures, and other        non-micro-optic elements to implement light-field imaging        sensing arrays;    -   Use of LED or OLED array as both a display and non-contact        (spatially-separated) lensless camera;    -   Use of transparent optoelectronic elements to implement stacked        wavelength-selective light-sensing elements analogous to Stacked        Organic Light Emitting Diodes (SOLEDs);    -   Use of bandgap responses to implement color image sensing arrays        without the use of filters, diffraction gratings, or other        wavelength-selective optical elements;    -   Curved image sensor arrays;    -   Bent and flexible image sensor arrays.

Corresponding novel mathematical and signal flow structures arranged tobe implemented by signal processing algorithms, and other novel systemelements and method steps also configurations comprised by theinventor's earlier patent families and used in the inventor's presentpatent application include for example but are not limited to:

-   -   Image formation signal processing:    -   Color separation signal processing;    -   Use of Moore-Penrose pseudo-inverse or other generalized        inverses to provide statistical robustness from over-specified        measurement data;    -   Use of Moore-Penrose pseudo-inverse or other generalized        inverses to provide spatial robustness (using over-specified        measurement data) against damage or occultation of isolated        sensor elements or groups of sensor elements;    -   Use of Graphics Processing Unit (“GPU”) for image formation        (particularly applicable when camera is integrated with a        display sharing the same multiplexing environment).

The inventor's early inventions (as taught in the afore-cited patents)and present invention competes favorably with other lenslesscomputational imaging approaches on their own terms in a number of ways,for example:

-   -   Significant computational efficiency advantage because no        transform domain conversions are required; images are recovered        by matrix multiplication of a long vector measurement vector by        a pre-computed matrix, facilitating video-rate decoding    -   Far greater light capture efficiency (incoming light loss can be        nearly zero, while coded apertures typically by their very        nature invoke a 45%-50% minimum loss of incoming light);    -   Cost reduction via printing (readily including printed interface        electronics);    -   Can be far thinner;    -   Can be far larger.

The inventor's early inventions (as taught in the afore-cited patents)and present invention provides a number of features not available fromother lensless computational imaging approaches, including but notlimited to:

-   -   No separation distance between the sensor and any vignetting or        aperturing array, allowing thinner image sensor;    -   No need for any vignetting or aperturing array if the        photosensing elements natively have adequate angular occultation        or angular selectively;    -   Vignetting or aperturing can be done at the same spatial        separation as individual photosensing pixels;    -   No special frequency domain requirements on any vignetting or        aperturing array;    -   Any vignetting or aperturing array can include        internally-reflective structures and have arbitrarily thin walls        to preclude light loss;    -   Any vignetting or aperturing array can be also arranged to        facilitate predictable or and/or reproducible surface plasmon        propagation to selected light sensors comprised by the light        sensor array in a manner that further reduces light loss;    -   Truly arbitrary image sensor size;    -   Flat or arbitrarily-curved image sensor shape;    -   Distributed image sensing;    -   Angular diversity/redundancy advantage;    -   Enveloping imaging, contact imaging (including local        illumination in contact imaging);    -   “Seeing skin”—rigid, flexible, deformable, manipulatable;    -   Integrated in a visual light-emitting display;    -   Overlay on provided surfaces;    -   Self-illuminating;    -   Zero separation distance focus;    -   Curved-surface contact focus;    -   Can provide one or more simultaneous computaionally-controlled        focus (mixed focus);    -   Can provide one or more simultaneous computationally-controlled        viewpoint(s);    -   Can provide one more simultaneous computationally-controlled        stereo/3D live imaging;    -   Can provide full-color and enhanced (meta-RGB) color image        capture;    -   Can include IR and UV capabilities;    -   Can include multiple-wavelength spectroscopic capabilities        without diffraction-grading or prism optics;    -   Can be integrated into a visual display.

FIG. 2 depicts an illustrative representational view of the confluenceof the expanded features and capabilities taught in the inventor's 1999patent family. This depiction and the elements therein are intended asonly illustrative and representative and does not provide or suggest acomprehensive or exhaustive listing, structure, or characterization.

Similarly, FIG. 3 depicts an illustrative representational view of theconfluence of the expanded features and capabilities taught in theinventor's 2008 patent family. This depiction and the elements thereinare intended as only illustrative and representative and does notprovide or suggest a comprehensive or exhaustive listing, structure, orcharacterization. Additional patent families

A 2008-2009 patent family of the present inventor contributed addtionalimplementations, features, and applications that include use as adisplay and a touch and touch-gesture user interface. A 2010 patentfamily and 2011 patent family of the present inventor contributedadditional implementations, features, and applications that include usenot only as a display touch user interface, and touch-gesture userinterface but also as a lensless imaging camera, a touch andtouch-gesture user interface, free-space hand gesture interface,document scanner, fingerprint sensor, secure information exchange.Although not all of the applications, arrangements, and configurationsof those 2008-2009, 2010, and 2011 patent families are explicitlyconsidered in the present patent application, the technologies, systems,and methods described the present patent application are in various waysdirectly applicable and to the applications, arrangements, andconfigurations described in those patent families.

A 2009-2010 patent family, and a 2013 patent family of the presentinventor contributed the addition of optical tomography capabilitiesemploying controlled light sources is also noted in FIG. 3. Although notall of the applications, arrangements, and configurations of those2009-2010 and 2013 patent families are explicitly considered in thepresent patent application, the technologies, systems, and methodsdescribed the present patent application are in various ways directlyapplicable and to the applications, arrangements, and configurationsdescribed in those patent families.

FIG. 4 depicts an illustrative representational view of the confluenceof the expanded features and capabilities associated with the presentinvention. This depiction and the elements therein are intended as onlyillustrative and representative and does not provide or suggest acomprehensive or exhaustive listing, structure, or characterization.

Relations to and Developments in Related Technologies

A brief review of the following related concepts and technologies arenext provided:

-   -   A. Lensless Coded Aperture Imaging;    -   B. Lens-Based Light-Field Imaging;    -   C. Use of LEDs as Light Sensors and Light Sensing Arrays;    -   D. Flexible Cameras and Transparent Image Sensors;    -   E. Transparent Electronics;    -   F. Organic Semiconductors and Organic Optoelectronics;    -   G. Printed Electronics and Optoelectronics;    -   H. Flexible and Bendable electronics;    -   I. Flexible and Bendable optoelectronics.        A summarizing functional- and timeline-comparison table is then        presented.        A. Relations to Lensless Coded Aperture Imaging

Coded apertures are planar, binary-valued(partially-opaque/partially-transmitting) optical masks gratings, grids,etc. positioned in front of an image sensor array and designed to caststructured shadows that permit mathematic calculations characterizingand permitting the imaging of incoming radiation fields. Originallydeveloped for high-energy photon (x-rays, gamma rays, and other classesof high-energy non-visible wavelength photons) radiation-imaging thatcannot be focused by lenses of curved mirrors, the beginnings of codedaperture imaging date back to at least 1968 [P62]. A number of codedaperture telescopes use this high-energy imaging approach for imagingastronomical X-ray ad gamma ray sources.

Partially predating and later developing in parallel with the inventor'scomprehensive lensless light-field imaging program (beginning with theinventor's 1999 patent family) is the related and nowrecently-popularized (2011-2016) coded aperture lensless imaging(recently termed a ‘Lensless “Computational Renaissance’” [P6]). Codedaperture imaging appears to have continued to develop in the exclusivecontext of high-energy photon non-visible (x-rays, gamma rays, etc.)radiation-imaging for decades but developing a rich mathematical theory(see for example [P31], [P32], [P63], [P64]) relating to the theory ofinformation codes and the relatively “flat” spectral properties of theoptical Modulation Transfer Function (discussed in part in variousearlier papers but see [P45]) when the coded aperture employs varioustypes of “Uniformly Redundant Array” codes (“URA”, “MURA,” etc.),variations or alternatives to these, or indeed randomly-generatedpatterns. At this writing it is not yet clear when attention to thesecoded aperture imaging approaches were first adapted for use invisible-light imaging, but at this writing it does not appear this wasexplored or discussed in the literature before 2000. There was some workcirca 1973 involving shadow casting and coded masks relating toholography [B8]. Further brief historical treatment and reviews oftechnology developments in coded aperture lensless imaging are providedin [B1], [P5], and [P65].

By 2006 various forms of computer-controlled spatial-light modulatorswere being used to implement the optical coded aperture function [P34],[P35], [P47], notably using the LCD of an LCD display screen as part ofthe MIT “BiDi Screen” [P34]. The MIT “BiDi Screen” also featureddistance ranging obtained from the coded aperture, described earlier in[P68]. Although the MIT “BiDi Screen” included coded aperture imaging,the images produced were not focused when any non-proximate distancefrom the screen.

As will be discussed later, most of these systems formulate the imagerecovery transformation as (1) an ill-posed, usually regularized,inverse problem and/or (2) a spectral-method transform problem.

Broader views of computations imaging have subsequently to appear in theliterature (see for example [P66]) which migrate the “optical coding”paradigm/abstraction to other contexts, for example wavefront coding,including a lens, sensor-plane coding, etc). In addition, there havealso been several relatively recent systems replacing coded apertureswith other types of lensless optical elements:

-   -   The 2008 Rice University [P48] and later 2013 Bell Labs [P59],        [P32], [P69], [P71], [P72] “single pixel” lensless camera        approach uses a computer-controlled aperture or micro-mirror        spatial light modulator to sequentially implement a        time-multiplexed series of coded aperture images. These        approaches can include sub-Nyquist rate “compressive sampling”        which can also be applied more broadly than optical sensing: se        for example [P60].    -   The post-2010 lensless imaging work at Cornell, later adopted to        create ultraminiature imagers and pre-commercialized by Rambus        [P4] (and winning best-paper award at the 2013 SensorComm        Conference), replaced the coded aperture with various types of        admirable intricate-design spiral-arm diffraction elements whose        optical Modulation Transfer Function possesses no spectral zeros        (as explicitly anticipated in the inventor's 1999 patent        family). The resulting diffracted light is measured by a small        but conventional CMOS imaging sensor. Although the available        Rambus papers and presentation slides describe various means of        numerical image recovery, inversion by numerical division by the        Fourier transform optical Modulation Transfer Function in the        Fourier numerical domain has been described as the preferred        method in an attended public presentation (again as explicitly        taught in the inventor's 1999 patent family). The Rambus system        also makes distance range measurements in keeping with similar        abilities of coded aperture imaging [P34], [P47]. The spiral-arm        diffraction element/CMOS imaging sensor approach was later        admirably extended to longer wavelengths through use of a        microbolometer to implement a miniature lensless far field 8-14        μm thermal-wavelength infrared computational imager [P61].    -   Hitachi announced a lensless light-field imaging camera,        targeted for 2018, that employs Moire patterns rather than a        coded aperture [P11], [P12], [P13], [P14].    -   CalTech announced an early-stage computational imaging camera        employing phase array methods [P39].    -   A University of Utah project utilizes a bare commercial CMOS        image sensor set a large distance away from a large “low”        resolution LED-display, using pixel-by-pixel training sequences        (as taught in a 2008 inventor's patent family) and recovering        the image as a regularized ill-posed inverse problem [P70].

Although the coded aperture imaging area and the three alternativesdescribed above each have their own “world,” it is possible to create anoverarching framework that includes all of these. As will be discussed,it is the inventor's view that the inventor's comprehensive lenslesslight-field imaging program (beginning with the inventor's 1999 patentfamily) arguably if not straightforwardly includes and provides aframework admitting most of these in at least some sense, as well asincluding the many other original innovations from the inventor'scomprehensive lensless light-field imaging program). For other selectiveperspectives and far more historical and contextual information see forexample [P6], [P66]. From an even broader view, all such approaches canbe abstracted into the notion of “computational imaging” from whichvarious fundamental principles can be explored; for example'see [P66],[P73].

As to further how the above compare and sequence over time with respectto the inventor's comprehensive lensless light-field imaging program,FIG. 5 depicts a more detailed view of the inventor's comprehensivelensless light-field imaging program (beginning with the inventor's 1999patent family) and includes recently-popularized coded-aperture lenslessimaging. FIG. 6 depicts a functional “timeline” view of lenslessimaging, including both the inventor's comprehensive lenslesslight-field imaging program (beginning with the inventor's 1999 patentfamily) is and recently-popularized (2011-2016) coded-aperture lenslessimaging stemming from radiation-imaging work dating from 1968 [P62].Additionally, FIG. 7 includes in its vertical time-line depictionrepresentative literature in lensless imaging with respect to theinventor's comprehensive lensless light-field imaging program.

As an additional note, since the fate of many captured images is to becompressed by image compression algorithms comprising at least somelinear transformation operations, the coded aperture could conceptuallybe modified to impose additional coding functions, in particular thoseuseful in compressing an image suitable for image decompression on theviewing or applications side. This has been shown to be possible andconsiderable work has been done in this area; see for example.

B. Relations to Lens-Based Light-Field Imaging

A light-field is most generally a 5-dimensional vector functionrepresentation of physical arrangement of directional light paths andintensities at each point in a space of optical propagation. The ideasdate back to Faraday but was named and formalized in 1936 by AndreyGershun. That said, the concept of a light field camera dates back tothe 1908 “Integral Photograph” work and proposals by Gabriel Lippmann,winner that same year of the Nobel Prize for the invention of colorphotography (also known for being the predictor of the now widelyemployed converse piezoelectric effect and inventor of the telescopeposition compensating coelostat).

A Stanford team including Ren Ng formalized a light-field camera using a(“plenoptic”) microlens-array technique [P40], leading not longthereafter to the well-respected Lytro refocusable light-field camera[P41], Toshiba subsequently announce a 2013 refocusable light-field OEMcamera module product [P42] using miniaturized similar technology.Recent (2016) developments in this area implement light-field imagingwithout the use of microlenses by employing layers of “optical” sensorsinstead [P43].

The inventor's 1999 patent family taught lensless computational imagingwith light-field capabilities and (albeit for color imaging sensing)layered optical sensors. Also, as indicated in the previous subsection,coded aperture image sensors are typically capable of performing aslight-field cameras (see for example [P67]).

It is notated notion, description, mathematical treatment, andmeasurement, of light-fields, and image rendering from them, have otherhistorical and contemporary threads. A 2006 survey of light-fieldimaging from a simplified 4-dimensional representation computationalimaging viewpoint employed in Image Based Rendering (IBR),computer-graphics fly-bys and related applications is presented in[P46]. Light-fields (and their analogs in acoustics, seismology, andenergy fields) are also in various forms are inherently and thusfundamentally relevant to at least wave-field inversion and wave-fieldInverse Source Problems (ISPs), tomography, holography, broader ImageBased Rendering (IBR) applications, and 3D graphics rendering, for a‘unified’ treatment regarding imaging, wavefield inversion andtomography see for example the book by Devaney [B3]. Additionally thereare various other methods for measuring and sampling empiricallight-fields; a few examples are described in the book by Zhang and Chen[B4].

Although presented before, FIG. 7 includes in its vertical time-linedepiction representative literature in light-field imaging with respectto the inventor's comprehensive lensless light-field imaging program.

C. Use of LEDs as Light Sensors and Light Sensing Arrays

A number of earlier U.S. Patents and U.S. Patent Applications discussvarious aspects of using LED and OLED arrays used in variouscombinations or sequences of light sensing and light-emitting modes andin one manor or another the integration of light sensing andlight-emitting semiconductors in a common display panel, control panel,or image reader. Some of these employ time-multiplexed operating modes,some of these spatially interleave light sensing and light-emittingsemiconductor elements, but none of these teach use as visual imagingcamera.

U.S. Pat. No. 4,424,524 by Daniele (filed 1982) teaches a linear arrayof LEDs that selectively function as light emitters and light sensorsfor line-scanning a document on a rotating drum. No image display isinvolved.

-   -   U.S. Pat. No. 4,692,739 by Dorn teaches use of LEDs to form user        operator panel control elements used both for receiving data        from an operator to change the logic state of a device and for        displaying the entered data back to the operator; current is        selectively applied to an LED to display the data and        alternately a photo-current produced by the LED is sensed by        detecting the fall-off in the photo-current caused by the        operator coveting the light-emitting diode. No image display or        capture is involved.    -   U.S. Pat. No. 5,424,855 by Nakamura teaches an array of LEDs        controlled so as to emit light in a write mode and sense light        in a read mode wherein each LED is alternately charged for a        first interval, then allowed to discharge by flow of        photocurrent for a second interval,    -   U.S. Pat. No. 5,929,845 by Wei teaches OLED array with        individual OLEDs multiplexed between emitting and sensing modes        (see text portion of the document, column 2 lines 4-17, and the        preceding paragraph spanning columns 1 and 2).    -   U.S. Pat. No. 7,598,949 by Han teaches an “optical touch sensor”        using LED-array as real-time photodiodes and co-integrated        light-emitters. In the “preferred embodiment” he uses an array        of discrete RGB LEDs, using (higher-energy/shorter-wavelength)        blue LED elements for light emission and red LED elements as        photodiodes (rather than time-multiplexing between emitting and        sensing modes), and alludes OLEDs and printable manufacturing        methods. A video demonstration (12 MB; viewable with MS Media        Player and other populate viewers) is downloadable from        http://mrl.nyu.edu/˜jhan/ledtouch/index.html (visited Jul. 3,        2017).    -   U.S. Pat. No. 7,859,526 by Konicek teaches an LED array with        individual LEDs multiplexed between emitting and sensing modes.    -   U.S. Pat. No. 8,026,879 by Booth teaches a somewhat different        “optical touch sensor” and provides discussion of OLEDs as        light-emitter and light-sensors (see abstract) but different use        of photosensing.    -   U.S. Pat. No. 8,890,850 by Chung also teaches a related “optical        touch sensor” and provides discussion of OLEDs as light-emitter        and light-sensors; in particular note FIGS. 4-6.    -   Abandoned U.S. Patent Application 2009/0256810 by Pasquariello        teaches a related “optical touch sensor” and provides some        discussion of multiplexing LEDs between emitting and sensing        modes.

These U.S. Patents discuss possible applications as camera but do notteach image formation:

-   -   The series U.S. Patents by Rostoker including U.S. Pat. Nos.        5,340,978, 5,519,205, 5,529,936, 5,734,155, 5,760,834, and        5,811,320.    -   U.S. Pat. No. 7,535,468 by Uy;    -   U.S. Pat. No. 6,787,810 by Choi et al.

Although presented before, FIG. 7 includes in its vertical time-linedepiction representative patents discussed in this section with respectto the inventor's comprehensive lensless light-field imaging program.

D. Flexible Cameras and Transparent Image Sensors

The limited work that has been done regarding flexible cameras has beenlargely in the enabling optics area. Many points regarding the value,radical enablement, and some applications of flexible cameras have beenprovided in papers and press releases stemming from projects at ColumbiaUniversity [P37], [P38] involving work on bendable and deformablemini-lens arrays, elastic optics, and associated internal opticalcompensational adaptation for those. These efforts appeal for the needfor the development of flexible image sensors; the prototyping workemploys traditional (color) camera sensors and camera lenses.

Another approach to flexible cameras, as well as flexible image sensorsand transparent image sensors, involves grid of flexible light-sensingfibers that direct light to remotely-located conventional camera element[P53].

Another third approach, to flexible cameras, as well as flexible imagesensors and transparent image sensors, underway in Austria [P28] also isdirected to enabling optics; this effort uses novel optics andluminescent concentrator thin transparent film to gather light from anarea of a transparent bendable surface and direct it to the edges of thetransparent bendable surface where it is provided to photodiode arraysat those edges. The images are monochrome. As to flexible image sensors,a flexible large-area photodetector array arranged as an image sensoremploying organic photodiodes (discussed below) has been reported in2008 [P54] where image-sensing capabilities have been demonstrated byprojecting an image using external image projection equipment. Morerecently, IMEC has made several pre-commercialization developmentsannounced in 2013 [P55] that overlap with the inventor's patents filedmany years earlier.

Although presented before, FIG. 7 includes in its vertical time-linedepiction representative literature in flexible cameras with respect tothe inventor's comprehensive lensless light-field imaging program.

E. Transparent Electronics

Developments in transparent electronics arguably began in earnest withthe discoveries, adaptations, and refinements of transparent conductormaterials (see for example [P56]. Various transparent passive electroniccomponents were subsequently developed and are important, but a keydevelopment was the invention of the first transparent thin-filmtransistor (TTFT) announced in 2003 ([B16] p. 1). TTFTs are presentlywidely used in display technology and can be fabricated by variousmeans, including spin-deposition and printing using ink-jet or otherprinting methods. Information on transparent electronic materials can befound in the book by Wagner, Keszler, and Presley [B16] (see Chapter 4)as well as information on transparent resistors ([B16] section 5.2.1),transparent capacitors ([B16] sections 5.2.2 and 5.3.4), transparentinductors ([B16] section 5.2.3), transparent PN diodes ([B16] section5.3.1), transparent MIS (Metal-Insulator-Semiconductor diodes ([B16]section 5.3.1), and transparent thin-film transistors (TTFT) ([B16]section 5.4). Additional information and applications are provided, forexample, in the book by Facchetti and Marks [B17].

F. Organic Semiconductors and Organic Optoelectronics

Closely related with the areas of transparent electronics, printedelectronics and flexible electronics is the area of organicsemiconductors and organic optoelectronics. Organic semiconductormaterials facilitate many aspects of transparent electronics, printedelectronics and flexible electronics, for example (a) replacing the bandgap employed in traditional crystalline semiconductors with the energyband transition between highest-occupied molecular orbitals andlowest-unoccupied molecular orbitals and (b) replacing the crystallattice structure of traditional crystalline semiconductors with thestructures of polymers. There are many other aspects of organicsemiconductors besides these. An introductory discussion of organicsemiconductor materials is provided for example in Chapter 2 of the 2004book by Gamota, Brazis, Kalyanasundaram, and Zhang [B22] and otherperspectives of organic semiconductor are provided in the 2013 bookedited by Cantatore [B23], although dozens of suitable and morecontemporary books and journal publications abound. One of manyimportant aspects is that organic semiconductor materials canfacilitating the use of solution-based (“ink”) printing fabrication (seefor example [B24]) and other techniques applicable to deposit on curvedsurfaces and large area surfaces and which facilitate flexible/bendableactive electronics. Other important aspects of organic semiconductormaterials include transparent capabilities and incorporation of a widerange of new optoelectronic capabilities.

A major commercial and technology aspect of organic electronics resultedfrom the development of an organic optoelectronic element known as theOrganic Light Emitting Diode (OLED). Attributions are made that the OLEDwas discovered at Kodak when researching solar cells (see for example[B20] section 3.3) and Kodak was a leader in this area for manysubsequent years. Combining with thin-film transistors, active-matrixOLED-array displays became commercially available and were used inmobile phones. Early active-matrix OLED-array displays suffered fromvarious problems and limitations, but the underlying materials, devices,system designs, and manufacturing techniques are yielding constraintimprovement, and every year or so new major products appear orannounced. One example is the curved-screen OLED television sets, andthe used of another generation of OLED displays have been announced forforthcoming new mass-market mobile phone products. Flexible OLEDdisplays have been repeatedly demonstrated at the annual ConsumerElectronics Shows for many consecutive years, and as will be consideredagain later, a bendable OLED display was included in a product-conceptpanoramic camera [P36]. Also as will be discussed later, transparentactive-matrix OLED-array displays and transparent OLED-arraypixel-addressing circuits using transparent thin-film transistors(TTFTs) and transparent capacitors, and transparent conductors have beendeveloped (see for example [B16] section 6.3.5; [B17] Chapter 12; [B18];[B19] section 8.2 [B23] Chapter 3.

OLED-array displays can be fabricated by printing (for example usingsemiconducting, conducting, insulative, and resistive inks) ornon-printed methods non-as discussion of non-printed fabrication methodscan be found in [B19] chapter 3, sections 6.1 and section 6.3).Developments in materials and fabrication techniques also createextension to size (including use of OLED-array tiling; see for example[B19] section 8.3), and degrees of curvature (for example a dome-shapedOLED display [P10]. Flat panel imager addressing circuits employingthin-film transistors and PIN or MIS light sensors are also known; seefor example [B18] sections 1.2, 2.2.1, 3.1, 3.2, 5.2, 6.1, and 6.2.

Attention is now is directed towards organic photodiode. Organicphotodiodes are widely viewed as providing an enabling route to newdevices and new applications that were previously impossible and likelyto remain outside the reach of conventional semiconductors. Althoughthere remains great devoted favoritism in the image sensor community forcrystalline semiconductor photodiodes monolithically-integrated withCMOS electronics (drawing on belief structures and early performancemetrics), organic photodiodes are rapidly gaining immense validation andradically expanding interest. As stated in the opening of [P22]:“Powerful, inexpensive and even flexible when they need to be, organicphotodiodes are a promising alternative to silicon-basedphotodetectors.” More detail as to this is provided in the summarizingremarks in the opening of [P2] “Organic photodiodes (OPDs) are now beinginvestigated for existing imaging technologies, as their properties makethem interesting candidates for these applications. OPDs offer cheaperprocessing methods, devices that are light, flexible and compatible withlarge (or small) areas, and the ability to tune the photophysical andoptoelectronic properties—both at a material and device level . . . withtheir performance now reaching point that they are beginning to rivaltheir inorganic counterparts in a number of performance criteriaincluding the linear dynamic range, detectivity, and color selectivity.”Additional important points are made in the opening remarks of [P17]:There are growing opportunities and demands for image sensors thatproduce higher-resolution images, even in low-light conditions.Increasing the light input areas through 3D architecture within the samepixel size can be an effective solution to address this issue. Organicphotodiodes (OPDs) that possess wavelength selectivity can allow foradvancements in this regard. Further important remarks are provided inthe opening words of [P30]: “Organic semiconductors hold the promise forlarge-area, low-cost image sensors and monolithically integratedphotonic microsystems . . . published structures of organic photodiodesoffer high external quantum efficiencies (EQE) of up to 76% . . . wereport on organic photodiodes with state-of-the-art EQE of 70% at 0 Vbias, an on/off current ratio of 106 . . . , dark current densitiesbelow 10 nA/cm2 . . . , and a lifetime of at least 3000 h . . . .” Othergeneral discussion of organic photodiodes can be found in manyreferences, for example [B21], [P21], [P27].

Like OLEDs and often using essentially the same materials, organicphotodiodes can readily be transparent [P20] or semitransparent [P19],fabricated via printing (see for example [P15], [P74]), and flexible(see for example [P2], [P54]). They can deliver high-sensitivity (seefor example [P30], [P74]), overall high-performance (see for example[P2], [P17], [P19], [P22], [P30]). They can also include avalanche andphotomultiplication (see for example [P18]). Additionally, likecrystalline semiconductor LEDs when used as light sensors, and have beenaffirmed to provide wavelength selective light-sensing properties (seefor example [P2], [P17], [P18], [P19], [P22], [P55]) that forgo the needfor optical filters (as pointed out years many prior in several of theinventor's patents) in, for example, color imaging.

Many efforts have been directed towards combining OLED arrays withvarious types of photosensors to create sensors for biomedicalapplications. Many of these combine OLED with silicon-basedphotodetectors (see for example [P16], [P23]), but there have beenrecent efforts involving combining OLEDs with organic photodiodes (seefor example [P7], [P8], [P9]). As to the enabling value of suchintegrations, [P23] states “Point-of-care molecular diagnostics canprovide efficient and cost-effective medical care, and they have thepotential to fundamentally change our approach to global health.However, most existing approaches are not scalable to include multiplebiomarkers. As a solution, we have combined commercial flat panel OLEDdisplay technology with protein microarray technology to enablehigh-density fluorescent, programmable, multiplexed biorecognition in acompact and disposable configuration with clinical-level sensitivity.”

As with crystalline semiconductor photosensors, increased performancefor some applications can often be obtained by creating photo-sensitivetransistors which, in effect, replace an electrically-responsivecontrolling input of a transistor with light-responsive controllinginput. Accordingly, there is active work in the area of organicphototransistors and organic phototransistor arrays (see for example[P25], [P27]), including for envisioned use as image detectors [P29].Like organic photodiodes, organic phototransistors can also bewavelength selective (see for example [P24],[P26]), high performance(see for example [P24], [P25], [P26], [P29]), and flexible (see forexample [P25], [P57]).

In addition to the aforedescribed transparent organic photodiode,transparent organic phototransistor and active matrix interface circuitsfor them, transparent charge-coupled devices are also known (see forexample [B16] section 6.3.6).

G. Printed Electronics and Optoelectronics

Printed electronics and optoelectronics have already been mentionedthrough many of the above sections. Information regarding generalmaterials for printed electronics can be found in a number ofpublications, for example [B17] p. 54, [B24], [B25], and [B26]. Organicsemiconductor materials suitable for printing can be found in a numberof publications, for example ([B22] Chapter 2). Printed electronicsmanufacturing processes can be found in a number of publications, forexample [B22] Chapter 3. OLED-array display printed fabrication isdiscussed in, for example, [B19] section 6.2 Printed Organic Photodiodesare discussed in, for example, featuring high-performance [P74] andcommercial availability [P15]. The important prospects for printableCMOS circuitry are discussed for example in [B16] p. 44; [B23] p. 124,and [P58].

H. Flexible and Bendable Electronics

Flexible and bendable electronics have already been mentioned throughmany of the above sections. Information regarding general materials forprinted electronics can be found in a number of publications (see forexample [B27], [B28], [B29], [B31], [B32], [P51]). General applicationsare discussed in [B27], [B28], [B29], and large area applications arediscussed in [B30]. Fabrication by printing methods are discussed inmany references (see for example [P51], [P52]), and performanceimprovements are frequently announced (see for example [P50]). Theexpected wide-acceptance of flexible electronics has been discussed in[P49]. The IOPscience multidisciplinary journal Flexible and PrintedElectronics™ publishes cutting edge search across all aspects of printedplastic, flexible, stretchable, and conformable electronics.

I. Flexible and Bendable Optoelectronics

Flexible and bendable electronics have already been mentioned throughmany of the above sections. Information regarding general materials forprinted electronics can be found in a number of publications (see forexample [B26]). Flexible TFTs for flexible OLED displays are discussedin for example [B19] section 8.1 and [P75]. High performance flexibleorganic photodiode arrays are discussed in for example [P54] and [P75].High performance flexible organic phototransistors arrays are discussedin [P25]. Large-area flexible organic photodiodes sheet image scannersare discussed in [P75]. A prototype for a commercial (panoramic camera)product employing a flexible OLED display, (c) conformation deformationsensing is described in [P36]. Although not using flexibleoptoelectonics, the related work on transparent, flexible, scalable anddisposable image sensors using thin film luminescent concentrators ispresented in [P28].

J. Summarizing Functional- And Timeline-Comparison Table

Although presented earlier, FIG. 5 depicts a summarizing functional viewof the inventor's comprehensive lensless light-field imaging program(beginning with the inventor's 1999 patent family), and FIG. 7 depicts acomparative timeline table of representative patents and literature withrespect to the inventor's comprehensive lensless light-field imagingprogram.

SUMMARY OF THE INVENTION

For purposes of summarizing, certain aspects, advantages, and novelfeatures are described herein. Not all such advantages can be achievedin accordance with any one particular embodiment. Thus, the disclosedsubject matter can be embodied or carried out in a manner that achievesor optimizes one advantage or group of advantages without achieving alladvantages as taught or suggested herein.

The invention provides for a rigid or flexible surface to be configuredto implement a lensless light-field sensor, producing electrical signalsthat can be used in real time, or stored and later retrieved, andprovided to a computational inverse model algorithm executing oncomputational hardware comprising one or more computing elements so asto implement a lensless light-field camera.

In another aspect of the invention, a rigid surface is configured toadditionally function as a housing and thus operate as a “seeinghousing”.

In another aspect of the invention, a rigid surface is configured toadditionally function as a protective plate and thus operate as a“seeing armor”.

In another aspect of the invention, a rigid surface is configured toadditionally function as an attachable tile and thus operate as a“seeing tile”.

In another aspect of the invention, a rigid surface is configured toadditionally function as an attachable film and thus operate as a“seeing film”.

In another aspect of the invention, a flexible surface is configured toadditionally function as an attachable film and thus operate as a“seeing film”.

In another aspect of the invention, a flexible surface is configured toadditionally function as a garment and thus operate as a “seeinggarment”.

In another aspect of the invention, a flexible surface is configured toadditionally function as a shroud and thus operate as a “seeing shroud”.

In another aspect of the invention, a flexible surface is configured toadditionally function as an enveloping skin and thus operate as a“seeing skin”.

In another aspect of the invention, the rigid or flexible surface issmall in size.

In another aspect of the invention, the rigid or flexible surface islarge in size.

In another aspect of the invention, the rigid or flexible surface isflat.

In another aspect of the invention, the rigid or flexible surface iscurved.

In another aspect of the invention, the rigid or flexible surface isrendered as a polytope.

In another aspect of the invention, the rigid or flexible surface isrendered as a dome.

In another aspect of the invention, the rigid or flexible surface isrendered as a part of a sphere.

In another aspect of the invention, the rigid or flexible surface isrendered as a part of a spheroid.

In another aspect of the invention, the rigid or flexible surface isrendered as a sphere.

In another aspect of the invention, the rigid or flexible surface isrendered as a spheroid.

In another aspect of the invention, the rigid or flexible surface istransparent.

In another aspect of the invention, the rigid or flexible surface istranslucent.

In another aspect of the invention, the rigid or flexible surface isopaque.

In another aspect of the invention, the rigid or flexible surfaceperforms contact sensing.

In another aspect of the invention, the rigid or flexible surface isconfigured to perform contact image sensing with near-zero separationdistance.

In another aspect of the invention, the rigid or flexible surface isconfigured to perform contact image sensing with zero separationdistance.

In another aspect of the invention, the rigid or flexible surfaceperforms distributed optical imaging.

In another aspect of the invention, the rigid or flexible surfaceperforms distributed optical sensing.

In another aspect of the invention, the rigid or flexible surfaceperforms image sensing of ultraviolet light.

In another aspect of the invention, the rigid or flexible surfaceperforms image sensing of infrared light.

In another aspect of the invention, the rigid or flexible surfaceperforms image sensing of selected ranges of visible color light.

In another aspect of the invention, the rigid or flexible surfaceperforms imaging.

In another aspect of the invention, the rigid or flexible surfaceperforms distributed chemical sensing employing optical chemical sensingproperties of at least one material.

In another aspect of the invention, the rigid or flexible surfaceperforms distributed radiation sensing employing optical radiationsensing properties of at least one material.

In another aspect of the invention, the rigid or flexible surfaceperforms distributed magnetic field sensing employing optical magneticfield sensing properties of at least one material.

In another aspect of the invention, the rigid or flexible surface isconfigured to emit light.

In another aspect of the invention, the rigid or flexible surface isconfigured to operate as a light-emitting display.

In another aspect of the invention, the rigid or flexible surface isconfigured to operate as a selectively self-illuminating contact imagingsensor.

In another aspect of the invention, the computational inverse modelalgorithm is configured to provide variable focusing.

In another aspect of the invention, the computational inverse modelalgorithm is configured to mixed depth-of-field focusing.

In another aspect of the invention, the computational inverse modelalgorithm is configured to implement a viewpoint with a controllablelocation.

In another aspect of the invention, the computational inverse modelalgorithm is configured to implement a plurality of viewpoints, eachviewpoint having a separately controllable location.

In another aspect of the invention, the computational inverse modelalgorithm is configured to provide pairs of outputs so as to function asa stereoscopic camera.

In another aspect of the invention, the computational inverse modelalgorithm is configured to capture a panoramic view.

In another aspect of the invention, the computational inverse modelalgorithm is configured to capture a 360-degree view.

In another aspect of the invention, the computational inverse modelalgorithm is configured to capture a partial spherical view.

In another aspect of the invention, the computational inverse modelalgorithm is configured to capture a full spherical view.

In another aspect of the invention, the rigid or flexible surface isconfigured to perform enveloping image sensing with near-zero separationdistance.

In another aspect of the invention, the rigid or flexible surface isconfigured to perform contact enveloping sensing with zero separationdistance.

In another aspect of the invention, the rigid or flexible surface isconfigured to operate as a selectively self-illuminating envelopingimaging sensor.

In another aspect of the invention, the computational inverse modelalgorithm is configured to operate at slow-frame video rates.

In another aspect of the invention, the computational inverse modelalgorithm is configured to operate at conventional video rates.

In another aspect of the invention, the computational inverse modelalgorithm and computational hardware is configured to operate athigh-speed video rates.

In another aspect of the invention, a lensless light-field imagingsystem comprising

-   -   An array of light sensing elements, each light-sensing element        comprising a light-sensing area and configured to generate an        electrical photocurrent responsive to the amplitude of incoming        light striking the light-sensing surface, each light-sensing        surface arranged to experience angularly-varying sensitivity        responsive to the direction of each path of incoming light        striking the light-sensing surface,    -   First electronics for interfacing the array of light sensing        elements, the electronics configured to provide a plurality of        voltage levels, each voltage level responsive to a specific        light-sensing element in the array of light sensing elements,    -   Second electronics for converting each of the plurality of        voltage levels into a corresponding electronically-represented        digital number, the result comprising a plurality of        electronically-represented digital numbers, and    -   An algorithm for executing on a computational processor, the        algorithm for computing a two-dimensional image representation        from plurality of electronically-represented digital numbers,        the two-dimensional image representation corresponding to        portion of a focused image at a particular separation distance        value measured perpendicular to the light-sensing surface of a        particular light sensing element in the array of light sensing        elements, there being a plurality of separation distance values,    -   Wherein each of the digital numbers are responsive to the        amplitude of incoming light striking the light-sensing surface        of a unique associated light sensing element in the array of        light sensing elements and a plurality of focused image        portions, and    -   Wherein the plurality of separation distance values are not        appreciably the same numeric value.

In another aspect of the invention, the light sensing elements of thearray of light sensing elements are oriented in space to form a curvedsurface.

In another aspect of the invention, spatial positions of the pluralityof focused image portions form a planar surface.

In another aspect of the invention, the light sensing elements of thearray of light sensing elements are oriented in space to form a planarsurface.

In another aspect of the invention, the spatial positions of theplurality of focused image portions form a curved surface.

In another aspect of the invention, the light sensing elements of thearray of light sensing elements are oriented in space to form a curvedsurface and the spatial positions of the plurality of focused imageportions form a curved surface.

In another aspect of the invention, the algorithm is controlled by a atleast one separation distance parameter.

In another aspect of the invention, the algorithm is controlled by aplurality of localized separation distance parameters.

In another aspect of the invention, the first electronics comprisesmultiplexing electronics.

In another aspect of the invention, the first electronics comprises atleast one transimpedance amplifier circuit.

In another aspect of the invention, the light sensing elements compriseorganic photodiodes.

In another aspect of the invention, the light sensing elements compriseorganic light emitting diodes.

In another aspect of the invention, the light sensing elements compriseorganic diodes that are co-optimized for both light emission and lightsensing.

In another aspect of the invention, the light sensing elements arearranged to emit light for some interval of time.

In another aspect of the invention, the light sensing elements arearranged to emit light for some interval of time under the control ofthe first electronics.

In another aspect of the invention, the angularly-varying sensitivity ofthe light sensing elements results at least in part from the structureof the light sensing elements.

In another aspect of the invention, the angularly-varying sensitivity ofthe light sensing elements results at least in part from a structureattached to the array of light sensing elements.

In another aspect of the invention, the array of light sensing elementsis fabricated by a printing process.

In another aspect of the invention, the stricture attached to the arrayof light sensing elements is fabricated by a printing process.

In another aspect of the invention, the structure attached to the arrayof light sensing elements comprises segregated optical paths.

In another aspect of the invention, the segregated optical paths arecreated by separating surfaces.

In another aspect of the invention, the separating surfaces are at leastpartially-reflective.

In another aspect of the invention, the separating surfaces areconfigured to facilitate surface plasmon propagation.

In another aspect of the invention, at least one of the light sensingelements is color selective.

In another aspect of the invention, color selective property resultsfrom a band gap property of a semiconductor device element comprised bythe at least one light sensor.

In another aspect of the invention, the algorithm comprises arraymultiplication of numerical values responsive to the plurality ofelectronically-represented digital numbers.

In another aspect of the invention, the algorithm comprises arraymultiplication of numerical values obtained from the calculation of ageneralized inverse matrix.

In another aspect of the invention, the algorithm comprises arraymultiplication of numerical values obtained from an interpolation.

In another aspect of the invention, the algorithm comprises arraymultiplication of numerical values obtained from a predictive analyticalmodel.

In another aspect of the invention, the algorithm comprises arraymultiplication of numerical values derived from a predictive analyticalmodel.

In another aspect of the invention, the algorithm comprises arraymultiplication of numerical values derived from empirical measurements.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features and advantages of the presentinvention will become more apparent upon consideration of the followingdescription of preferred embodiments taken in conjunction with theaccompanying drawing figures, wherein:

FIG. 1 depicts an example conceptual view of the underlying principlesof the invention, facilitating a wide range of implementation methodsand architectures.

FIG. 2 depicts an illustrative representational view of the confluenceof the expanded features and capabilities taught in the inventor's 1999patent family.

FIG. 3 depicts an illustrative representational view of the confluenceof the expanded features and capabilities taught in the inventor's 2008patent family. The 2009 addition of optical tomography capabilities isalso noted.

FIG. 4 depicts an illustrative representational view of the confluenceof the expanded features and capabilities associated with the presentinvention. This depiction and the elements therein are intended as onlyillustrative and representative and does not provide or suggest acomprehensive or exhaustive listing, structure, or characterization.

FIG. 5 depicts a more detailed view of the inventor's comprehensivelensless light-field imaging program (beginning with the inventor's 1999patent family) is and recently-popularized coded-aperture lenslessimaging.

FIG. 6 depicts a functional “timeline” view of non-pinhole lenslessimaging, including both the inventor's comprehensive lenslesslight-field imaging program (beginning with the inventor's 1999 patentfamily) is and recently-popularized (2011-2016) coded-aperture lenslessimaging (recently termed a ‘Lensless “Computational Renaissance”’[P6]stemming from radiation-imaging work dating from 1968[ P62].

FIG. 7 depicts a timeline of representative technology-related patentsand literature with respect to the inventor's comprehensive lenslesslight-field imaging program.

FIG. 8 a, adapted from FIG. 42 of the present inventor's U.S. Pat. No.8,830,375 and related cases, depicts a vector space of trade-offs forsemiconducting diode-junction devices.

FIG. 8 b, adapted from FIG. 42 of the present inventor's U.S. Pat. No.8,830,375 and related cases, depicts an adaptation of FIG. 8a whereindifferent optimizations are used for implementing single functiondiode-junction devices such as (but not limited to) switching diodesversus light-emitting diodes (LEDs) versus photodiodes.

FIG. 9 a, adapted from the figure available on the internet athttps://en.wikibooks.org/wiki/Introduction_to_Inorganic_Chemistry/Electronic_Properties_of_Materials:_Superconductors_and_Semiconductors#/media/File:PnJunction-E,PNG as retrieved Jul. 3, 2017 (top portion), depicts a representation ofthe active carrier flow of a forward biased switching diode wherein, bydesign, current-flow directional switching functions are optimized andlight-emission and light-detection capabilities of PN junctions aresuppressed.

FIG. 9 b, adapted from the figure available on the internet athttps://en.wikibooks.org/wiki/Introduction_to_Inorganic_Chemistry/Electronic_Properties_of_Materials:_Superconductors_and_Semiconductors#/media/File:PnJunction-E,PNG as retrieved Jul. 3, 2017 (middle portion), depicts the blockedcarrier flow of a reversed biased situation for the switching diodedepicted in FIG. 9 a.

FIG. 9 c, adapted from the figure available on the internet athttps//en.wikibooks.org/wiki/Introduction_to_Inorganic_Chemistry/Electronic_Properties_of_Materials:_Superconductors_and_Semiconductors#/media/File:PnJunction-E,PNGas retrieved Jul. 3, 2017 (bottom portion), depicts an energy-bandrepresentation of a switching diode wherein by design, current-flowdirectional switching functions are optimized and light-emission andlight-detection capabilities of PN junctions are suppressed.

FIG. 9 d, adapted from the image available athttp//www.learnabout-electronics.org/Semiconductors/diodes_23.php asvisited on Jun. 20, 2017, depicts a representation of the physicalconstruction of a switching diode wherein, by design, current-flowdirectional switching functions are optimized and light-emission andlight-detection capabilities of PN junctions are suppressed.

FIG. 10 a, adapted from the top portion of a figure available on theinternet athttps//en.wikipedia.org/wiki/Light-emitting_diode#/media/File:PnJunction-LED-E.svgas retrieved Jul. 3, 2017 depicts a carrier-process representation of anoperating (inorganic or organic) conducting PN junction light-emittingdiode (LED).

FIG. 10 b, adapted from the bottom portion of a figure available on theinternet athttps://en.wikipedia.org/wiki/Light-emitting_diodes#/media/File:PnJunction-LED-E.svgas retrieved Jul. 3, 2017, depicts an energy-transition representationof an operating (inorganic or organic) semiconducting PN junctionlight-emitting diode (LED).

FIG. 11 a, adapted from Figure 4.7.1 of the on-line notes “Principles ofSemiconductor Devices” by B. Van Zeghbroeck, 2011, available athttps://ecee,colorado.edu/˜bart/book/book/chapter4/ch4_7.htm asretrieved Jul. 3, 2017, depicts an abstracted structural representationof an example (inorganic or organic) simple (“simple-heterostructure”)semiconducting PN junction light-emitting diode (LED).

FIG. 11 b, adapted from Figure 7.1 of the on-line table of figuresavailable on the internet athttps://www.ecse.rpi.edu/˜schubert/Light-Emitting-Diodes-dot-org/chap07/chap07.htmas retrieved Jul. 3, 2017, depicts an abstracted structuralrepresentation of an example (inorganic or organic) more complexdouble-heterostructure semiconducting PN junction light-emitting diode(LED), here effectively configured as a two-PN junction sandwich. FIG.12a (adapted from G. Gu, G. Parthasarathy, P. Burrows, T. Tian, I. Hill,A. Kahn, S. Forrest, “Transparent stacked organic light emittingdevices. I. Design principles and transparent compound electrodes,”Journal of Applied Physics, October 1999, vol. 86 no. 8, pp. 4067-4075)depicts an example high-level structure of a three-color transparentStacked OLED (“SOLED”) element.

FIG. 12b (also adapted from G. Gu, G. Parthasarathy, P. Burrows, T.Tian, I. Hill, A. Kahn, S. Forrest, “Transparent stacked organic lightemitting devices. I. Design principles and transparent compoundelectrodes,” Journal of Applied Physics, October 1999, vol. 86 no. 8,pp. 4067-4075) depicts a more detailed structure of a three-colortransparent SOLED element.

FIG. 13 a, depicts an example energy-transition representation of anoperating (inorganic or organic) simple semiconducting PN junctionphotodiode.

FIG. 13 b, simplified and adapted from the first two figures in“Comparison of waveguide avalanche photodiodes with InP and InAIAsmultiplication layer for 25 Gb/s operation” by J. Xiang, and Y. Zhao,Optical Engineering, 53(4), published Apr. 28, 2014, available athttp://opticalengineering.spiedigitallibrary.org/article.aspx?articleid=1867195as retrieved Jul. 3, 2017, depicts an example structural representationof an example simple layered-structure PIN (inorganic or organic) simplesemiconducting PN junction photodiode.

FIG. 14 a, adapted from FIG. 2 of U.S. Pat. No. 7,202,102 “DopedAbsorption for Enhanced Responsivity for High Speed Photodiodes” to J.Yao, depicts a combined energy/structure representation of a morespecialized example layered-structure avalanche, semiconducting PNjunction photodiode.

FIG. 14 b, adapted from the first two figures in “Comparison ofwaveguide avalanche photodiodes with InP and InAIAs multiplication layerfor 25 Gb/s operation” by J. Xiang and Y. Zhao, Optical Engineering,53(4), published Apr. 28, 2014, available athttp://opticalengineering.spiedigitallibrary.org/article.aspx?articleid=1867195as retrieved Jul. 3, 2017, depicts an example structural representationof an example layered-structure avalanche semiconducting PN junctionphotodiode.

FIG. 15a depicts material science and fabrication relationships among(1) transparent/non- transparent electronics and optoelectronics, (2)flexible/non-flexible electronics and optoelectronics, (3)printed/non-printed electronics and optoelectronics, and (4)organic/non-organic electronics and optoelectronics.

FIG. 15b provides a version of FIG. 15a where certain types of theelectronics and optoelectronics are marked with asterisks (*) to signifyfunctional contributions to various aspects of the present invention.

FIG. 16 a, adapted from [P5], depicts a schematic representation of thearrangements and intended operational light paths for a pinhole camera.

FIG. 16 b, adapted from [P5], depicts a schematic representation of thearrangements and intended operational light paths for a (simplified orsingle-lens) lens-based camera.

FIG. 16 c, adapted from [P5], depicts a schematic representation of thearrangements and intended operational light paths for a mask-basedcamera, such as those discussed in [P62]-[P67].

FIG. 16d depicts to a schematic representation of some aspects of thepresent invention and the inventor's more comprehensive lenslesslight-field imaging program.

FIG. 17 depicts an array of parallel-oriented vignetting cavities; thebottom of each cavity can comprise or direct isolated light tolight-sensing structure.

FIG. 18, adapted from FIG. 12 of the present inventor's U.S. Pat. No.8,816,263 and related cases, illustrates a simplified view of how avignette structure can limit the range of incident angles at which raysof light within a light field are able to reach the surface of thelight-sensing element within a vignetting structure covering alight-sensing element. (Importantly, reflective effects within thevignette and diffraction effects are not illustrated.)

FIG. 19, composited and adapted from FIGS. 8 and 9 a through 9 b of thepresent inventor's U.S. Pat. No. 8,830,375 and related cases,illustrates a simplified view of the process by which the degree ofvignette overlap increases as separation between the object in the sceneand its distance from the micro-optic structure and light sensor arrayincreases and how the degree of vignette overlap increases from 0% tovalues approaching 100% as the separation distance between a sceneobject and the micro-optic structure and light sensor array increases.(Importantly, reflective effects within the vignette and diffractioneffects are not illustrated.)

FIGS. 20a through 20c depict illustrative representations of reflectionand scattering effects within a vignette. (Importantly, diffractioneffects are not illustrated.)

FIG. 21, adapted from FIG. 11b of the present inventor's U.S. Pat. No.8,816,263 and related cases, depicts an array of parallel-orientedinstances of alternating short” light-sensing structures and “tall”,each parallel-oriented instance alternately staggered to createvignetting cavities surrounded by the sides of neighboring “tall”structures (which in some implementations can be light-emitting), thebottom of each cavity comprising a “short” light-sensing structure. Insome implementations, the “tall” structures can be light-emitting.

FIGS. 22a through 22c depict differing illustrative 3-dimensional viewsof a plane the containing the sensing surface of a planar image sensorarrangement, and extending spatially in front of the planar image sensora coordinate grid defining numerically quantizing regions on an incominglight field that can be observed by the planar image sensor arrangement.Depending on the directional capabilities of the planar image sensorarrangement, the shape of the observable light field can have adifferent shape than the illustrative rectangular parallelepiped.

FIG. 23a depicts an example spatial quantization of a light fieldextending spatially in front of the planar image sensor into a latticedistinct of indexable volume elements (voxels).

FIG. 23b depicts an example spatial quantization of the light fieldvoxel lattice of FIG. 23a by representing the aggregate oflight-emission, light reflection, and/or light propagation within thevoxel as (1) having a composite quantitative value of light representingthe combined aggregate of light-emission, light reflection, and/or lightpropagation within the volume of voxel which is (2) concentrated at apoint in the interior of the voxel.

FIG. 24 depicts a pair of illustrative 3-dimensional views of an examplearrangement comprising a planar array of (emitted and/or reflected)light source elements and a parallel planar array of light-sensingelements, and a spatially-quantized light-field representation betweenthe planes. The roles of planar array of light source elements and aparallel array of light-sensing elements can be interchanged.

FIGS. 25a and 25b depict a pair of illustrative 3-dimensional views ofan example variation of the arrangement depicted in FIG. 24 wherein aplanar array of (emitted and/or reflected) light source elements and aplanar array of light-sensing elements are not parallel planes. Theroles of planar array of light source elements and a parallel array oflight-sensing elements can be interchanged.

FIG. 26a depicts another illustrative 3-dimensional view of an examplevariation of the arrangement depicted in FIGS. 25a and 25b wherein thedihedral angle between the planes is farther from parallel. The roles ofplanar array of light source elements and a parallel array oflight-sensing elements can be interchanged.

FIG. 26b depicts an illustrative 3-dimensional view of another exampleof the arrangements depicted in FIGS. 25 a, 25 b, and 26 a wherein thedihedral angle between the planes is sloped in two dimensions. The rolesof planar array of light source elements and parallel array oflight-sensing elements can be interchanged.

FIGS. 27a and 27b depict a pair of illustrative 3-dimensional views ofan example of a non-planar curved surface and a planar surface with aspatially-quantized light-field representation between the two surfaces.

FIGS. 28a and 28b depict a pair of illustrative 3-dimensional views of avariation on FIGS. 27a and 27b featuring different example non-planarcurved surface.

FIG. 29a depicts an illustrative example of non-planar curved surfacesensor and non-planar curved surface object with a spatially-quantizedlight-field representation between the two surfaces. Either or both ofthe curved surfaces can be configured to be a camera.

FIG. 29b depicts an example cross-section of a (rigid or flexible)curved imaging surface, for example as may be fabricated by printing orother deposition fabrication processes.

FIGS. 30a through 30c depict a variety of illustrative 3-dimensionalviews of an example variation of the arrangement depicted in FIGS. 24,25 a, 25 b, 26 a, and 25 b wherein the array of (emitted and/orreflected) light source elements are split among a plurality of smallerparallel planes at different separation distances from the planar arrayof light-sensing elements. Depending on the spatial arrangement, someportions of some of the smaller parallel planes can be observationallyocculted from some regions of the planar surface. The roles of theplurality of planar arrays of light source elements and a parallel arrayof light-sensing elements can be interchanged.

FIGS. 31a and 31b depict a variety of illustrative 3-dimensional viewsof an example variation of the arrangement depicted in FIGS. 24, 25 a,25 b, 26 a, and 25 b wherein the array of (emitted and/or reflected)light source elements are distributed over a connected group of planesat least one parallel to the planar array of light-sensing elements.

FIGS. 32a through 32c depict a variety of illustrative 3-dimensionalviews of an example wherein the array of (emitted and/or reflected)light source elements are distributed over a complex collection ofconnected and disconnected planes, some of which are parallel to theplanar array of light-sensing elements, and some of whichobservationally occulted others from some regions of the planar surfaceby being situated directly in front of others.

FIGS. 33a end 33 b depict example inward-directed or outward-directedsensor-pixel lattice locations distributed on a rigid or elastic curvedconvex-shaped surface.

FIGS. 34a through 34d depicts examples of pairs of curved andsharply-angled surfaces, one of the pair inside the other of that pair.In any of the arrangements depicted, at least one of the inner surfaceand the outer surface can be a camera arranged to view the othersurface. As considered elsewhere, the camera can be configured toprovide self-illumination.

FIGS. 35a through 35c depict illustrative examples of bumpy and/orpitted sensor surfaces that can provide angular diversity. Sucharrangements can also be used to provide sensor robustness vie spatialdiversity, to provide directed angle-orientation viewing, and to provideother types of functions. These can be combined with the generousrecovery capabilities described in the mathematical treatment to follow,and enhanced further by the statistical corrections obtainable using theMoore-Penrose pseudo-inverse, to provide immense imaging robustness to awide range of degradation and partial occultation effects.

FIG. 36 depicts example correspondences between a physical opticalarrangement comprising an optical process (including for examplevignetting optics and free-space separation should the image sensor notbe in contact with the actual source image) and a mathematical model ofthat physical optical arrangement (transforming an actual image arraydata to measured image array data by a numerical model of the opticalprocess.

FIG. 37 depicts an illustrative example of Mathematical Recovery of anapproximate representation of the actual Image from Measured Image ArrayData obtained by operating on the Measured Image Array Data by aNumerical Inverse of the Model of the Optical Process as depicted inFIG. 36.

FIG. 38, adapted from FIG. 2b of the present inventor's U.S. Pat. No.8,830,375 and related cases, depicts an exemplary embodiment comprisinga micro-optic structure, a light sensor array, an image formation signalprocessing operation and an optional additional subsequent imageprocessing operations, herein the micro-optic structure and light sensorarray are grouped into a first subsystem, and the image formation signalprocessing operation and subsequent image processing operations aregrouped into a second subsystem. As discussed in the present inventor'sU.S. Pat. No. 8,830,375 and related cases, various other arrangementsare possible and provided for by aspects of the invention.

FIG. 39a depicts an example scheme wherein manufacturing, physical,optical, and mathematical considerations are used to create areproducible manufacturing design such that is adequate manufacturingtolerances are obtained an analytical predictive model can be used toproduce numerical models of the optical situations to be recoveredwithout the use of empirical measurements.

FIG. 39b depicts an example variation on the scheme presented in FIG.39a wherein post-manufacturing empirical measurements are used tofurther fine-calibrate the system performance of each particularmanufactured article.

FIG. 40 depicts an example representation of example serializationprocesses and de-serialization processes for image recovery from aninverse or pseudo-inverse model as provided for by the invention.

FIG. 41 depicts a representation of example serialization processestransforming a measured image produced by a light-field travellingthrough an optical structure (here a vignette array) at being measuredby a sensor array.

FIG. 42 depicts a representation of example empirical image-basis“training” sequence, or alternatively a collection ofpredictive-model-generated image-bases that directly populate a JK×NMmatrix providing a numerical model of the optical environment from whicha future image will later be recovered as provided for by aspects of theinvention.

FIG. 43a depicts a representation of example image recovery processusing an inverse square-matrix representing an approximate inverse modelas provided for by the invention.

FIG. 43b depicts a representation of example image recovery processusing a generalized-inverse or pseudo-inverse matrix representing anapproximate pseudo-inverse underspecified model as provided for by theinvention.

FIG. 43c depicts a representation of example image recovery processusing a generalized-inverse or pseudo-inverse matrix representing anapproximate pseudo-inverse overspecified model as provided for by theinvention.

FIG. 44a depicts a simple computational approach for image recovery asprovided for by the invention.

FIG. 44b depicts an example representation of a far morenumerically-complex spectral or transform computation (which numericallyamounts to additional basis rotation transformation steps) as would beused in spectral or transform methods.

FIG. 44c depicts a comparison of the more direct computational approachdepicted in FIG. 44a and the approach depicted in FIG. 44 b,demonstrating comparative reasons to reject the far morenumerically-complex spectral or transform computational approachedrepresented in FIG. 44 b.

FIG. 45 depicts the use of classical ill-posed inverse-problemregularization methods as employed in the Rambus [P4] and RiceUniversity “Flat Cam” [P5] implementations as well as other codedaperture imaging implementations. These classical ill-posedinverse-problem regularization methods are widely used in many areas butalso have an established role in optical systems design and analysis;see for example [B5] and [B6].

FIG. 46 depicts an abstract representation of an “Identity structure”within a (necessarily-sparse) Identity 4-tensor.

FIG. 47 depicts an abstract representation of a dilation around the“Identity structure” within a (necessarily-sparse) Identity 4-tensor.

FIG. 48 depicts a ((7×7)×(7×7)) “matrix-of-matrices” representation of a(7×7×7×7) Identity 4-tensor as abstracted in FIG. 46.

FIG. 49 depicts an example ((7×7)×(7×7)) “matrix-of-matrices”representation of the dilation around the “Identity structure” of a(7×7×7×7) 4-tensor as abstracted in FIG. 46.

FIG. 50 depicts an abstract representation of the trade-off betweenSpace/Inverse-Space/Spatial Frequency Localization Methods and thesparcity of numerical model tensors, matrices, and their inverses. Thiscan be used, for example, in designing vignetting and aperture arrays.

FIGS. 51a through 51e depict how the sparcity of an example numericalmodel matrix serializing a numerical model 4-tensor degrades as theimage source moves farther and farther from the image sensor (usingnumerical models at each distances predicted by an analytical geometricpredictive model provided for by the invention).

FIG. 52 depicts an example ((7×7)×(7×7)) “matrix-of-matrices”representation of the numerical model 4-tensor and example measuredimage (right) at zero separation distance from asingle-illuminated-pixel source image (left). As described elsewhere thesource image need not be on a parallel plane and can be distributedarbitrarily in space.

FIG. 53 depicts an example ((7×7)×(7×7)) “matrix-of-matrices”representation of the numerical model 4-tensor and example measuredimage (right) at a small non-zero separation distance from the samesingle-illuminated-pixel source image (left) as in FIG. 52.

FIG. 54 depicts an example ((7×7)×(7×7)) “matrix-of-matrices”representation of the numerical model 4-tensor and example measuredimage (right) at zero separation distance from a two-illuminated-pixelsource image (left). As described elsewhere, the source image need notbe on a parallel plane and can be distributed arbitrarily in space.

FIG. 55 depicts an example ((7×7)×(7×7)) “matrix-of-matrices”representation of the numerical model 4-tensor and example measuredimage (right) at a small non-zero separation distance from the sametwo-illuminated-pixel source image (left) as in FIG. 54.

FIG. 56 depicts an example ((7×7)×(7×7)) “matrix-of-matrices”representation of the numerical model 4-tensor and example measuredimage (right) at zero separation distance from a more closely-spacedtwo-illuminated-pixel source image (left). As described elsewhere, thesource image need not be on a parallel plane and can be distributedarbitrarily in space.

FIG. 57 depicts an example ((7×7)×(7×7)) “matrix-of-matrices”representation of the numerical model 4-tensor and example measuredimage (right) at a small non-zero separation distance from the same moreclosely-spaced two-illuminated-pixel source image (left) as in FIG. 56.

FIGS. 58a through 58c depict three of forty-nine steps of an exampleempirical training sequence as provided for by the invention. In animplementation of the invention, such a procedure could be performed fora selected collection of one or more separation distances between theimage sensor and the source image. As described elsewhere, the sourceimage need not be on a parallel plane and can be distributed arbitrarilyin the observable space.

FIG. 59 depicts an example polynomial fitting function interpolationmethod for interpolating the numerical model, itsinverse/pseudo-inverse, and/or other functions for separation distancesvalues lying between k empirically-trained separation distances. Asdescribed elsewhere, the source image need not be on a parallel planeand can be distributed arbitrarily in the observable space.

FIG. 60 depicts an example polynomial fitting function interpolationmethod for interpolating the numerical model, itsinverse/pseudo-inverse, and/or other functions for separation distancesvalues lying between k separation distances used by a predictive-model.As described elsewhere, the source image need not be on a parallel planeand can be distributed arbitrarily in the observable space.

FIG. 61 depicts an example of very poor curve-fit interpolation of thematrix elements of the numerical model between measure data distancesresulting from not having correct or sufficient terms in a modelpolynomial.

FIG. 62 depicts an example piecewise-linear interpolation of the matrixelements of the numerical model between k empirically-trained separationdistances.

FIG. 63a depicts how with very small separation distances and certainnon-alignment of source-pixel locations and sensor pixel locations somesensor pixel locations cannot receive light from proximate source-pixellocations, while FIG. 63b depicts how at slightly greater separationdistances this condition does not occur, such processes can give rise tothe rising and falling of selected curves in the example empiricaltraining data shown in the example of FIG. 61.

FIG. 64 depicts an example piecewise-linear interpolation method forinterpolating the numerical model, its inverse/pseudo-inverse, and/orother functions for separation distances values lying between kempirically-trained separation distances. As described elsewhere, thesource image need not be on a parallel plane and can be distributedarbitrarily in the observable space.

FIG. 65 depicts an example piecewise-linear interpolation method forinterpolating the numerical model, its inverse/pseudo-inverse, and/orother functions for separation distances values lying between kseparation distances used by a predictive-model. As described elsewhere,the source image need not be on a parallel plane and can be distributedarbitrarily in the observable space.

FIG. 66 depicts an abstract representation of the mathematical recoveryof image from measured image array data.

FIG. 67 depicts a variation on the representation of the mathematicalrecovery of image from measured image array data shown in FIG. 66wherein more measurements are made than needed, resulting in anover-specified collection of measurements that can be expected to leadto inconsistent calculation outcomes when different subgroups ofmeasurements are used to solve for the recovered image. As provided forby the invention, a Moore-Penrose pseudo-inverse operation gives aleast-squares fit to the expected inconsistent outcomes; this isdepicted in the bottom portion of the figure.

FIG. 68a through 68d depict example numerical model outcomes responsiveto a single-illuminated pixel as generated for various separationdistances by an example predictive analytical geometric model. Theparticular example predictive analytical geometric model used onlyaccounts for vignette occultation as calculated by simple ray-trancingand does not include the effects of vignette-internal reflections,vignette-internal scattering, vignette-aperture diffraction, surfaceplasmoid propagation, etc.

FIG. 69a depicts three example interactive quantization-effect modelingoutcomes wherein controllable quantization is artificially imposed onmeasurement data so as to predictively model and characterize theeffects of quantizing nonlinearities imposed by electronicdigital-to-analog converter processes on empirical training,predictive-model generated, and real-time measurements as they influencethe quality of image recovery.

FIG. 69b depicts three example interactive offset-effect modelingoutcomes wherein controllable measurement offset is artificially imposedon measurement data so as to predictively model and characterize theeffects of measurement offsets imposed by electronic digital-to-analogconverter processes on empirical training, predictive-model generated,and real-time measurements as they influence the quality of imagerecovery.

FIG. 69c depicts an example interactive noise-modeling control used tointroduce synthetically-generated noise and noise processes so as topredictively model and characterize its effects. The selection showncontrols additive Gaussian noise, but other noise processes associatedwith photodiodes can similarly be introduced.

FIGS. 70a through 70d depict example interactive modeling outcomesshowing the effect of noise, offset, and both these for a 32-step(5-bit) quantized (“DN” [B2]) measurement dynamic range.

FIGS. 71a through 71d depict example interactive modeling outcomesshowing the effect of noise, offset, and both these for a 64-step(6-bit) quantized “DN”) measurement dynamic range.

FIG. 72a through 72d depict example interactive modeling outcomesshowing the effect of noise, offset, and both these for a 128-step(7-bit) quantized (“DN”) measurement dynamic range.

FIGS. 73a through 73d depict example interactive modeling outcomesshowing the effect of noise, offset, and both these for a 140-step(slightly more than 7-bit) quantized (“DN”) measurement dynamic range.

FIG. 74a through 74d depict example interactive modeling outcomesshowing the effect of noise, offset, and both these for a 150-step (yetmore than 7-bit) quantized (“DN”) measurement dynamic range.

FIG. 75 a, adapted from [B7] Figure C2.5.16(a), depicts a first examplepixel-cell multiplexed-addressing circuit for an individual OLED withinan OLED array that includes a dedicated light-coupled monitoringphotodiode for use in regulating the light output of the individual OLEDso as to prevent user-observed fading or other brightness-variationprocesses.

FIG. 75 b, adapted from [B7] Figure C2.5.16(b), depicts a second examplepixel-cell multiplexed-addressing circuit for an individual OLED withinan OLED array that includes a dedicated light-coupled monitoringphotodiode for use in regulating the light output of the individual OLEDso as to prevent user-observed fading or other brightness-variationprocesses.

FIG. 76 a, adapted from [B18], depicts an example pixel-cellmultiplexed-addressing circuit for an individual OLED within an OLEDarray with a monitoring feature.

FIG. 76 b, also adapted from [B18], depicts an example pixel-cellmultiplexed-addressing circuit for an isolated high-performancephotodiode or phototransistor within a high-performance photodiode orphototransistor array with a forced-measurement provision.

FIG. 77 depicts a simplified view of FIG. 1 showing only the signal andcomputational portion of FIG. 1 as will be useful in a subsequentdiscussion.

FIG. 78 depicts a variation of the arrangement represented in FIG. 77wherein the Inverse Model is rendered as a parameterized Inverse Modelwhich can be altered responsive to one or more provided parameters.

FIG. 79 depicts a variation of the arrangement represented in FIG. 78wherein the electrical signals and/or computational data produced by theOptical Sensor are in parallel provided in whole or selected (orselectable) part to a plurality of Inverse Models, each producing one ormore computationally-produced images responsive to the sensor data.

FIG. 80 depicts a variation of the arrangement represented in FIG. 78wherein the electrical signals and/or computational data produced by theOptical Sensor is handled by a computer or computational element (suchas a microprocessor GPU, DSP chip, ALU, FPLA, combination of two or morethese, pluralities of these, etc.) in some fashion that at least permitsthe electrical signals and/or computational data produced by the OpticalSensor to be stored as a file.

FIG. 81 depicts a variation of the arrangement represented in FIG. 78wherein the aforementioned handling by a computer or computationalelement is controlled in some manner by a control parameter.

FIG. 82a depicts a variation of the arrangement represented in FIG. 81wherein a plurality of stored files is created, for example withdifferent parameter values associated with each stored file.

FIG. 82b depicts an example arrangement wherein a stored is used by afixed Inverse Model to create a computationally-produced image.

FIG. 82c depicts an example arrangement wherein a stored file is used bya parameterized Inverse Model to create a computationally-producedimage, and further wherein parameter value(s) associated theparameterized Inverse Model are externally associated with each storedfile.

FIG. 82d depicts an example arrangement wherein a stored file is used bya parameterized Inverse Model to create a computationally-producedimage, and further wherein parameter value(s) associated theparameterized Inverse Model are derived from or obtained from the storedfile.

FIG. 83 depicts four example conformations of a particular bendable,flexible, and/or or pliable optical-sensor/optical-structure sheet eachconformation giving rise to an associated model for how a light field issensed in the context of reconstructing an image, and each model givingrise to its own associated inverse model. It is noted that, as dependenton the properties and limitations of the optical-structure, there can besmall blind spots in regions of sufficiently-high curvature.

FIG. 84a through FIG. 84d depict how various conformations can be usedto render a computationally derived image.

FIG. 85a depicts an example arrangement for anoptical-senor/optical-structure of fixed conformation.

FIG. 85b depicts an example arrangement for anoptical-senor/optical-structure of variable conformation, for exampleshould the surface bend, deform, hinge, expand, contract, etc.

FIG. 85c depicts a variation of FIG. 85b wherein the model and theInverse Model are parameterized.

FIG. 86a depicts an arrangement useful for using optical training tosense the present conformation of an optical-senor/optical-structure ofvariable conformation, for example should the surface bend, deform,hinge, expand, contract, etc.

FIG. 86b depicts an arrangement useful for using internal sensing meansto sense the present conformation of an optical-senor/optical-structureof variable conformation, for example should the surface bend, deform,hinge, expand, contract etc.

FIG. 86c depicts an arrangement useful for using information about amovable or changing support structure or contact arrangement to identifythe present conformation of an optical-senor/optical-structure ofvariable conformation, for example should the surface bend, deform,hinge, expand, contract, etc.

FIG. 86d depicts an arrangement useful for using external observationmean, for example such as one or more observing video camera(s) means tosense the present conformation of an optical-senor/optical-structure ofvariable conformation, for example should the surface bend, deform,hinge, expand, contract, etc.

FIG. 87a depicts an optical-senor/optical-structure of variableconformation for example should the surface bend, deform, hinge, expand,contract, etc. producing an output signal and/or output data.

FIG. 87b depicts a controllable variable-conformation material whoseshape/conformation can be controlled by externally-provide controlstimulus.

FIG. 87c depicts an optical-senor/optical-structure of variableconformation, for example should the surface bend, deform, hinge,expand, contract, etc. producing an output signal and/or output datafabricated on, with, or co-integrated with controllablevariable-conformation material whose shape/conformation can becontrolled by externally-provide control stimulus.

FIG. 88a depicts use of conformational sensing information to derive orcompute parameter values for a parameterized Inverse Model.

FIG. 88b depicts use of conformational control parameter information toderive or compute parameter values for a parameterized Inverse Model.

FIG. 88c depicts use of both conformational sensing information andconformational control parameter information to derive or computeparameter values for a parameterized Inverse Model.

DETAILED DESCRIPTION

In the following description, reference is made to the accompanyingdrawing figures which form a part hereof, and which show by way ofillustration specific embodiments of the invention. It is to beunderstood by those of ordinary skill in this technological field thatother embodiments may be utilized, and structural, electrical, as wellas procedural changes may be made without departing from the scope ofthe present invention.

In the following description, numerous specific details are set forth toprovide a thorough description of various embodiments. Certainembodiments may be practiced without these specific details or with somevariations in detail. In some instances, certain features are describedin less detail so as not to obscure other aspects. The level of detailassociated with each of the elements or features should not be construedto qualify the novelty or importance of one feature over the others.

FIG. 7 depicts a timeline of representative patents and literature andpatents in lensless imaging and light-field imaging with respect to theinventor's comprehensive lensless light-field imaging program.

FIG. 8 a, adapted from FIG. 42 of the present inventor's U.S. Pat. No.8,830,375 and related cases, depicts a vector space of trade-offs forsemiconducting diode-junction devices. As taught in the inventor'searlier patents, there are opportunities to, rather than exclusivelyoptimize for light-sensing or light-emission, to instead co-optimize forboth light-sensing and light-emission.

The present invention provides for co-optimizing of doping, electrodeconfigurations, structure, and other attributes for both light-sensingand light-emission, giving rise to entirely new kinds of semiconductoroptoelectronic elements and devices. Rapidly evolving organicsemiconductor material science methods, including polymer properties andmeta-material properties, can be used to improve quantum efficiency,noise performance, transparency, size requirements, electricalcharacteristics, etc. as well as facilitate useful manufacturingtechniques such as high-resolution printing. Additional features, suchas angular-selectivity and wavelength selectivity, can also be included.Additional structures, such as vignetting or aperturing arrays,reflective optical path walls to reduce incident light-loss, angulardiversity, curvature, flexibility, etc. can be co-integrated, and canfor example be designed to produce predictable reproducible opticalsensing behaviors. Exotic features, such as predictable or and/orreproducible surface plasmon propagation to selected light sensors tofurther reduce incoming light loss and use of quantum dots, can beincluded.

FIG. 8 b, adapted from FIG. 42 of the present inventor's U.S. Pat. No.8,830,375 and related cases, depicts an adaptation of FIG. 8a whereindifferent optimizations are used for implementing single functiondiode-junction devices such as (but not limited to) switching diodesversus light-emitting diodes (LEDS) versus photodiodes.

FIG. 9 a, adapted from the figure available on the internet athttps://en.wikibooks.org/wiki/Introduction_to_Inorganic_Chemistry/Electronic_Properties_of_Materials:_Superconductors_and_Semiconductors#/media/File:PnJunction-E.PNGas retrieved Jul. 3, 2017 (top portion), depicts a representation of theactive carrier flow of a forward biased switching diode wherein, bydesign, current-flow directional switching functions are optimized andlight mission and light-detection capabilities of PN junctions aresuppressed.

FIG. 9 c, adapted from the figure available the internet athttps://en.wikibooks.org/wiki/Introduction_to_Inorganic_Chemistry/Electronic_Properties_of_Materials:_Superconductors_and_Semiconductors#/media/File:PnJunction-E.PNGas retrieved Jul. 3, 2017 (middle portion), depicts the blocked carrierflow of a reversed biased situation for the switching diode depicted inFIG. 9 a.

FIG. 9 c, adapted from the figure available on the internet athttps://en.wikibooks.org/wiki/Introduction_to_Inorganic_Chemistry/Electronic_Properties_of_Materials:_Superconductors_and_Semiconductors#/media/File:PnJunction-E.PNGas retrieved Jul. 3, 2017 (bottom portion), depicts an energy-bandrepresentation of a switching diode wherein by design, current-flowdirectional switching functions are optimized and light-emission andlight-detection capabilities of PN junctions, are suppressed.

FIG. 9 d, adapted from the image available athttp://www.learnabout-electronics.org/Semiconductors/diodes_23.php asvisited on Jun. 20, 2017, depicts a representation of the physicalconstruction of a switching diode wherein, by design, current-flowdirectional switching functions are optimized and light-emission andlight-detection capabilities of PN junctions are suppressed.

FIG. 10 a, adapted from the top portion of a figure available on theinternet athttps://en.wikipedia.org/wiki/Light-emitting_diode#/media/File:PnJunction-LED-E.svgas retrieved Jul. 3, 2017, depicts a carrier-process representation ofan operating (inorganic or organic) semiconducting PN junctionlight-emitting diode (LED).

FIG. 10 b, adapted from the bottom portion of a figure available on theinternet athttps://en.wikipedia.org/wiki/Light-emitting_diode#/media/File:PnJunction-LED-E.svgas retrieved Jul. 3, 2017, depicts an energy-transition representationof an operating (inorganic or organic) semiconducting PN junctionlight-emitting diode (LED).

FIG. 11 a, adapted from Figure 4.7.1 of the on-line notes “Principles ofSemiconductor Devices” by B. Van Zeghbroeck, 2011, available athttps://ecee.colorado.edu/˜bart/book/book/chapter4/ch4_7.htm asretrieved Jul. 3, 2017, depicts an abstracted structural representationof an example (inorganic or organic) simple (“single-heterostructure”)semiconducting PN junction light-emitting diode (LED).

FIG. 11 b, adapted from Figure 7.1 of the on-line table of figuresavailable on the Internet athttps://www.ecse.rpi.edu/˜schubert/Light-Emitting-Diodes-dot-org/chap07/chap07.htmas retrieved Jul. 3, 2017, depicts an abstracted structuralrepresentation of an example (inorganic or organic) more complexdouble-heterostructure semiconducting PN junction light-emitting diode(LED), here effectively configured as a two-PN junction sandwich. Whencomponent layers are properly doped, a P-I-N(“P-type”/“Intrinsic”/“N-type”) structure is formed, confining chargecarriers into a “small” energy gap surrounded by abrupt energydiscontinuities that can be used to create a quantum well; the chargecarriers recombine in the “Intrinsic” region and emit photons withwavelengths defined by corresponding discrete permissible energytransitions.

FIG. 12a (adapted from G. Gu, Parthasarathy, P. Burrows, T. Tian, I.Hill, A. Kahn, S. Forrest, “Transparent stacked organic light emittingdevices. I. Design principles and transparent compound electrodes,”Journal of Applied Physics, October 1999, vol. 86 no. 8, pp. 4067-4075)depicts an example high-level structure of a three-color transparentStacked OLED (“SOLED”) element.

FIG. 12b (also adapted from G. Gu, G. Parthasarathy, P. Burrows, T.Tian, I. Hill, A. Kahn, S, Forrest, “Transparent stacked organic lightemitting devices. I, Design principles and transparent compoundelectrodes,” Journal of Applied Physics, October 1999, vol. 86 no. 8,pp. 4067-4075) depicts a more detailed structure of three-colortransparent SOLED element.

FIG. 13 a, depicts an example energy-transition representation of anoperating (inorganic or organic) simple semiconducting PN junctionphotodiode.

FIG. 13 b, simplified and adapted from the first two figures in“Comparison of waveguide avalanche photodiodes with InP and InAIAsmultiplication layer for 25 Gb/s operation” by J. Xiang and Y. Zhao,Optical Engineering, 53(4), published Apr. 28, 2014, available athttp://opticalengineering.spiedigitallibrary.org/article.aspx?articleid=1867195as retrieved Jul. 3, 2017, depicts an example structural representationof an example simple layered-structure PIN (inorganic or organic) simplesemiconducting PN junction photodiode.

FIG. 14 a, adapted from FIG. 2 of U.S. Pat. No. 7,202,102 “DopedAbsorption for Enhanced Responsivity for High Speed Photodiodes” to J.Yao, depicts a combined energy/structure representation of a morespecialized example layered-structure avalanche semiconducting PNjunction photodiode.

FIG. 14 b, adapted from the first two figures in “Comparison ofwaveguide avalanche photodiodes with InP and InAIAs multiplication layerfor 25 Gb/s operation” by J. Xiang and Y. Zhao, Optical Engineering,53(4). published Apr. 28, 2014, available athttp://opticalengineering.spiedigitallibrary.org/article.aspx?articleid=1867195as retrieved Jul. 3, 2017, depicts an example structural representationof an example layered-structure avalanche semiconducting PN junctionphotodiode.

FIG. 15a depicts material science and fabrication relationships among(1) transparent/non- transparent electronics and optoelectronics, (2)flexible/non-flexible electronics and optoelectronics, (3)printed/non-printed electronics and optoelectronics, and (4)organic/non-organic electronics and optoelectronics.

FIG. 15b provides a version of FIG. 15a where certain types of theelectronics and optoelectronics are marked with asterisks (*) to signifyfunctional contributions to various aspects of the present invention.

FIG. 16 a, adapted from [P5], depicts a schematic representation of thearrangements and intended operational light paths for a pinhole camera.The box represents a light-tight enclosure with a pinhole opening on theleft side that blocks much of the incoming light field (depicted asapproaching from the right) but permits transmission of narrow-diameterincoming light rays to enter the enclosure and travel through a regionof free-space so as to widen the light area to match that of a(typically rectangular) image sensor, film emulsion display surface,etc.

FIG. 16 b, adapted from [P5], depicts a schematic representation of thearrangements and intended operational light paths for a (simplified orsingle-lens) lens-based camera. The box represents a light-tightenclosure with a lens and supporting opening for the lens on the leftside that bends most rays of the incoming light field (depicted asapproaching from the right) for transmission and travel through a regionof free-space defined by the lens focal length and the lens law equationso as to create focused image of a selected depth-of-field onto a(typically rectangular) image sensor, film emulsion display surface,etc.

FIG. 16 c, adapted from [P5], depicts a schematic representation of thearrangements and intended operational light paths for a mask-basedcamera, such as those discussed in [P62]-[P67]. The relatively flatterbox represents a light-tight enclosure with a masked opening on the leftside that blocks some of the incoming light field (depicted asapproaching from the right) and permits transmission the remainingincoming light rays to enter the enclosure and travel through a shorterregion of free-space so as to widen the light area to match that of a(typically rectangular) image sensor.

FIG. 16d depicts to schematic representation of some aspects of thepresent invention and the inventor's more comprehensive lenslesslight-field imaging program. No free space is needed and any vignettingoptical structure can directly contact and/or be co-integrated orlayered upon (by deposition, printing, etc.) the image sensor surface.The optical width of such a vignetting optical structure can be assmall. as one light-sensing pixel in a light-sensing array, and such avignetting optical structure can (unlike a mask or the Rambus [P4]diffraction element) have a very simple structure.

The invention further provides for vignetting arrays, aperturing arrays,or other optical structures attached to, co-fabricated on, orco-fabricated with an array of light sensors to include for example,reflective optical path walls to reduce incident light-loss, angulardiversity, curvature, flexibility etc.

The invention further provides for vignetting arrays, aperturing arrays,or other optical structures attached to, co-fabricated on, orco-fabricated with an array of light sensors to be designed to producepredictable reproducible optical sensing behaviors. The inventionfurther provides for vignetting arrays, aperturing arrays, or otheroptical structures attached to, co-fabricated on, or co-fabricated withan array of light sensors to include or facilitate advancelight-processing features such a predictable or and/or reproduciblesurface plasmon propagation to selected light sensors to further reduceincoming light loss, use of quantum dots, etc.

Additionally the invention provides for each light-sensing pixel elementin a light-sensing array to comprise one or separatewavelength-selective light-sensing sub-elements, for example as taughtin the inventor's 1999 and 2008 patent families. In some implementationsthese sub-elements can be spatially adjacent and share the samevignetting or other light-structuring pathway. In other implementationsit is advantageous to stack two or more wavelength-selectivelight-sensing sub-elements in layers, analogous to Stacked Organic LightEmitting Diodes (SOLEDs) as discussed in the inventor's 1999 and 2008patent families. It is further noted that structures stacking layers oftwo or more wavelength-selective light-sensing sub-elements can bedesigned to limit or advantageously structure different vignettingeffects each wavelength-selective light-sensing sub-element willexperience at each particular depth in the layered stack. It is notedthat recent (2016) developments in this area implement light-fieldimaging (without the use of microlenses) employing layers of “optical”sensors [P43].

FIG. 17 depicts an array of parallel-oriented vignetting cavities; thebottom of each cavity can comprise or direct isolated light tolight-sensing structure.

FIG. 18, adapted from FIG. 12 of the present inventor's U.S. Pat. No.8,816,263 and related cases, illustrates a simplified view of how avignette structure can limit the range of incident angles at which raysof light within a light field are able to reach the surface of thelight-sensing element within a vignetting structure covering alight-sensing element. (Importantly, reflective effects within thevignette and diffraction effects are not illustrated.)

FIG. 19, composited and adapted from FIGS. 8 and 9 a through 9 b of thepresent inventor's U.S. Pat. No. 8,830,375 and related cases,illustrates a simplified view of the process by which the degree ofvignette overlap increases as separation between the object in the sceneand its distance from the micro-optic structure and light sensor arrayincreases and how the degree of vignette overlap increases from 0% tovalues approaching 100% as the separation distance between a sceneobject and the micro-optic structure and light sensor array increases,(Importantly, reflective effects within the vignette and diffractioneffects are not illustrated.)

FIGS. 20a through 20c depict illustrative representations of reflectionand scattering effects within a vignette. (Importantly, diffractioneffects are not illustrated.)

FIG. 21, adapted from FIG. 11b of the present inventor's U.S. Pat. No.8,816,263 and related cases, depicts an array of parallel-orientedinstances of alternating short” light-sensing structures and “tall”,each parallel-oriented instance alternately staggered to createvignetting cavities surrounded by the sides of neighboring “tall”structures (which in some implementations can be light-emitting), thebottom of each cavity comprising “short” light-sensing structure. Insome implementations, the “tall” structures can be light-emitting.

Light-Field Origins, Propagation, and Lensless-Light-Field Sensing

Returning to the depiction illustrated in FIG. 1, an Optical Scenecreates a Light-Field that is directed to an Optical Sensor which ispreceded by one or more Lensless Optical Structure(s) that in somemanner alters the light field in a predictable spatial manner. TheOptical Sensor produces (typically time-varying) electrical signalsand/or computational data responsive (instantly and/or within sometime-delay) to light incident to the surface or other substructure(s)within the Optical Sensor at any given moment.

In terms of the mathematical development above, objects or situationsproducing reflected, refracted, and/or light-emitted contributions tothe Light-Field can be represented in a spatially-quantized manner as alight-field source array.

FIGS. 22a through 22c depict differing illustrative 3-dimensional viewsof a plane the containing the sensing surface of a planar image sensorarrangement, and extending spatially in front of the planar image sensora coordinate grid defining numerically quantizing regions on an incominglight field that can be observed by the planar image sensor arrangement.Depending on the directional capabilities of the planar image sensorarrangement, the shape of the observable light field can have adifferent shape than the illustrative rectangular parallelepiped.

FIG. 23a depicts an example spatial quantization of a light fieldextending spatially in front of the planar image sensor into a latticeof distinct indexable volume elements (voxels).

FIG. 23b depicts an example spatial quantization of the light fieldvoxel lattice of FIG. 23a by representing the aggregate oflight-emission, light reflection, and/or light propagation within thevoxel as (1) having a composite quantitative value of light representingthe combined aggregate of light-emission, light reflection, and/or lightpropagation within the volume of voxel which is (2) concentrated at apoint in the interior of the voxel. If the light-field is indexed bywavelength or wavelength range, the composite quantitative value can berepresented as a function of an associated wavelength index orquantized-wavelength index. The point within each voxel in the lightfield can be used to define a spatially-quantized vector fieldrepresenting a physical spatially-continuous vector field (and/ornumerical representation thereof). Accordingly, composite quantitativevalue of light representing the combined aggregate of light-emission,light reflection, and/or light propagation within the volume of voxelcan further be represented as a function with a directional argument. Insuch a manner, the spatial and spectral (wavelength) aspects of aspatially (and if relevant, spectrally) quantized representation of aphysical light field can be computationally represented as amultiple-index array.

Case A: Fixed Separation Distance:

Although it will be shown that the constraints on this arrangement canbe extremely relaxed, in can be initially convenient to regard theobjects or situations producing contributions to the light-field aslying in a plane parallel to an image sensor plane, and thecontributions to the light-field comprising, a planar (for examplerectangular, other shapes explicitly admissible) array oflight-providing “light-source” spatially-quantized pixels, each“light-source” pixel emitting light that in various manners make theirway to a parallel spatially-separated image sensor plane. The imagesensor plane comprises a planar (for example rectangular, other shapesexplicitly admissible) array of light-providing spatially-quantized“light-sensing” pixels, these “light-sensing” pixels producing anelectrical signal that can be further processed. The discussion andcapabilities of this development explicitly include cases with zeroseparation distance between at least a planar array of light-providing“light-source” pixels and at least a planar array of “light-sensing”pixels.

As an illustration of Case A, FIG. 24 depicts a pair of illustrative3-dimensional views of an example arrangement comprising a planar arrayof (emitted and/or reflected) light source elements and a parallelplanar array of light-sensing elements, and a spatially-quantizedlight-field representation between the planes. Note the roles of planararray of light source elements and a parallel array of light-sensingelements can be interchanged.

Case B: Continuous Spatially-Varying Separation Distances —Non-ParallelPlanes:

Relaxing the constraints in Case A, the above-described planes of (a)the objects or situations producing contributions to the light-field and(b) the image sensor are not parallel but rather oriented at somenon-parallel and non-perpendicular dihedral angle. The resultinglight-field has separation mixed distances. The discussion andcapabilities of this development explicitly include cases with zeroseparation distance between at least a subset (even a 1-dimensional edgeor even a single point) of a planar array of light-providing“light-source” pixels and a subset (even a 1-dimensional edge or even asingle point) of at least a planar array of “light-sensing” pixels.

As an illustration of Case B, FIGS. 25a and 25b depict a pair ofillustrative 3-dimensional views of an example variation of thearrangement depicted in FIG. 24 wherein a planar array of (emittedand/or reflected) light source elements and a planar array oflight-sensing elements, are not parallel planes. Note the roles ofplanar array of light source elements and a parallel array oflight-sensing elements can be interchanged.

As another illustration of Case B, FIG. 26a depicts another illustrative3-dimensional view of an example variation of the arrangement depictedin FIGS. 25a and 25b wherein the dihedral angle between the planes isfarther from parallel. Note the roles of planar array of light sourceelements and a parallel array of light-sensing elements can beinterchanged.

As yet another illustration of Case B, FIG. 26b depicts an illustrative3-dimensional view of another example of the arrangements depicted inFIGS. 25 a, 25 b, and 26 a wherein the dihedral angle between the planesis sloped in two dimensions. Note the roles of planar array of lightsource elements and a parallel array of light-sensing elements can beinterchanged.

Example C: Continuous Spatially-Varying Separation Distances—CurvedSurfaces:

Relaxing the constraints in Case A in yet another way, one or both of(a) the objects or situations producing contributions to the light-fieldand/or (b) the image sensor reside on smoothly-curved non-planarsurface. The resulting light-field has separation mixed distances and insome cases possible occultation depending on variations in curvature.The discussion and capabilities of this development explicitly includecases with zero separation distance between at least a subset (even a1-dimensional edge or even a single point) of a planar array oflight-providing “light-source” pixels and a subset (even a 1-dimensionaledge or even a single point) of at least a planar array of“light-sensing” pixels.

As an illustration of Case C, FIGS. 27a and 27b depict a pair ofillustrative 3-dimensional views of an example of a non-planar curvedsurface and a planar surface with a spatially-quantized light-fieldrepresentation between the two surfaces. Note that conceptually that (1)the planar surface could comprise light-sensing elements that observethe non-planar curved surface which in this role comprises (reflectiveor emitting) light source elements, or (2) if the imaging surface can berendered as the depicted illustrative non-planar curved surface, thenon-planar curved surface could comprise light-sensing elements thatobserve the planar surface which in this role comprises (reflective oremitting) light source elements. It is noted that regions of thenon-planar curved surface that are convex or concave with respect to theplanar surface can be observationally occulted from some regions of theplanar surface.

As another illustration of Case C, FIGS. 28a and 28b depict a pair ofillustrative 3-dimensional views of a variation on FIGS. 27a and 27bfeaturing different example non-planar curved surface. Note here, too,that conceptually that (1) the planar surface could compriselight-sensing elements that observe the non-planar curved surface whichin this role comprises (reflective or emitting) light source elements,or (2) if the imaging surface can be rendered as the depictedillustrative non-planar curved surface, the non-planar curved surfacecould comprise light-sensing elements that observe the planar surfacewhich in this role comprises (reflective or emitting) light sourceelements. It is also noted that regions of the non-planar curved surfacethat are convex or concave with respect to the planar surface can beobservationally occulted from some regions of the planar surface.

As a variation of Case C, FIG. 29 depicts an illustrative examplecross-section of a non-planar (rigid or flexible) curved surface sensorand non-planar curved surface object with a spatially-quantizedlight-field representation between the two surfaces. Either or both ofthe curved surfaces can be configured to be a camera, and sucharrangements can be fabricated by printing or other depositionfabrication processes. Depending on the spatial arrangement, someportions of some of one of the curved surfaces can be observationallyocculted from some regions of the other curved surface Note the roles ofthe plurality of planar arrays of light source elements and a parallelarray of light-sensing elements can be interchanged.

Case D: Multiple Parallel Planes of Mixed Discrete Separation Distances:

Relaxing the constraints in Case A in still another way, either one orboth of (a) the objects or situations producing contributions to thelight-field and/or (b) the image sensor resides on more than one planarsurfaces but with various separation distances, typically a mix ofseparation distances. A resulting light-field comprises mixed separationdistances with abrupt changes in the separation distance.

As an illustration of Case D, FIGS. 30a through 30c depict a variety ofillustrative 3-dimensional views of an example variation of thearrangement depicted in FIGS. 24, 25 a, 25 b, 26 a, and 25 b wherein thearray of (emitted and/or reflected) light source elements are splitamong a plurality of smaller parallel planes at different separationdistances from the planar array of light-sensing elements. Depending onthe spatial arrangement, some portions of some of the smaller parallelplanes can be observationally occulted from some regions of the planarsurface. Note the roles of the plurality of planar arrays of lightsource elements and a parallel array of light-sensing elements can beinterchanged.

Case E: Combinations of at Least One Parallel Planes and at Least OneNon-Multiple Parallel Plane:

Generalizing in Case D further, one or more instances of the situationsof Case A and Case B are combined, resulting in a more complexlight-field. In many situations occultation of portions of light fieldscan occur, for example in cases where one non-transparent source arrayblocks at least a portion of the light emitted by another source arrayas viewed by (one or more of the) sensor array(s).

As an illustration of Case E, FIGS. 31a and 31b depict a variety ofillustrative 3-dimensional views of an example variation of thearrangement depicted in FIGS. 24 25 a, 25 b, 26 a, and 25 b wherein thearray of (emitted and/or reflected) light source elements aredistributed over a connected group of planes, at least one parallel tothe planar array of light-sensing elements. Depending on the spatialarrangement, some portions of some of the non-parallel planes can beobservationally occulted from some regions of the planar surface. Notethe roles of the plurality of planar arrays of light source elements anda parallel array of light-sensing elements can be interchanged.

Case F: More Complex Combinations of Mixed Discrete and ContinuousSpatially-Varying Separation Distances:

Generalizing in Case E further, one or more instances of the situationsof at least one of Case A and Case B are combined with at least oneinstance of the situation of Case C, resulting in yet a more complexlight-field. In many situations occultation of portions of light fieldscan occur, for example in cases where one non-transparent source arrayblocks at least a portion of the light emitted by another source arrayas viewed by (one or more of the) sensor array(s).

As an illustration of Case F, FIGS. 32a through 32c depict a variety ofillustrative 3-dimensional views of an example wherein the array of(emitted and/or reflected) light source elements are distributed over acomplex collection of connected and disconnected planes, some of whichare parallel to the planar array of light-sensing elements, and some ofwhich observationally occulted others from some regions of the planarsurface by being situated directly in front of others. Note the roles ofthe plurality, of planar arrays of light source elements and a parallelarray of light-sensing elements can be interchanged.

FIGS. 33a and 33b depict example inward-directed or outward-directedsensor-pixel lattice locations distributed on a rigid or elastic curvedconvex-shaped surface. As considered elsewhere, leveraging theco-integration of light-emitting and light-sensing elements as aught inthe inventor's 1999 patent family, the camera can be configured toprovide self-illumination, for example when used as anelastically-fitted cap that serves as a zero-separation-distancecontact-imaging camera monitoring the surface of an enclosed object. Asexplained elsewhere, since an arbitrary-shaped “focus-surface” can becomputationally defined, should the elastic cap only make contact withsome portions of an object enshrouded by such a (self-illuminating ifneeded) elastic cap, focused images of the entire encapsulated region ofthe non-occulted surface of the enshrouded object can be produced.

FIGS. 34a through 34d depicts examples of pairs of curved andsharply-angled surfaces, one of the pair inside the other of that pair.In any of the arrangements depicted, at least one of the inner surfaceand the outer surface can be a camera arranged to view the othersurface. As considered elsewhere the camera can be configured to provideself-illumination.

FIGS. 35a through 35c depict illustrative examples of bumpy and/orpitted sensor surfaces that can provide angular diversity. Sucharrangements can also be used to provide sensor robustness via spatialdiversity, to provide directed angle-orientation viewing, and to provideother types of functions.

FIG. 36 depicts example correspondences between a physical opticalarrangement comprising an optical process (including for examplevignetting optics and free-space separation should the image sensor notbe in contact with the actual source image) and a mathematical model ofthat physical optical arrangement (transforming an actual image arraydata to measured image array data by a numerical model of the opticalprocess.

FIG. 37 depicts an illustrative example of Mathematical Recovery of anapproximate representation of the actual Image from Measured Image ArrayData obtained by operating on the Measured Image Array Data by aNumerical Inverse of the Model of the Optical Process as depicted inFIG. 36.

FIG. 38, adapted from FIG. 2b of the present inventor's U.S. Pat. No.8,830,375 and related cases, depicts an exemplary embodiment comprisinga micro-optic structure, a light sensor array, an image formation signalprocessing operation and an optional additional subsequent imageprocessing operations, herein the micro-optic structure and light sensorarray are grouped into a first subsystem, and the formation signalprocessing operation and subsequent image processing operations aregrouped into a second subsystem. As discussed in the present inventor'sU.S. Pat. No. 8,830,375 and related cases, various other arrangementsare possible and provided for by aspects of the invention.

FIG. 39a depicts an example scheme wherein manufacturing, physical,optical, and mathematical considerations are used to create areproducible manufacturing design such that is adequate manufacturingtolerances are obtained an analytical predictive model can be used toproduce numerical models of the optical situations to be recoveredwithout the use of empirical measurements.

FIG. 39b depicts an example variation on the scheme presented in FIG.39a wherein post-manufacturing empirical measurements are used tofurther fine-calibrate the system performance of each particularmanufactured article.

FIG. 40 depicts an example representation of example serializationprocesses and de-serialization processes for image recovery from aninverse or pseudo-inverse model as provided for by the invention.

FIG. 41 depicts a representation of example serialization processestransforming a measured image produced by a light-field travellingthrough an optical structure (here a vignette array) at being measuredby a sensor array.

FIG. 42 depicts a representation of example empirical image-basis“training” sequence, or alternatively a collection ofpredictive-model-generated image-bases that directly populate a JK×NMmatrix providing a numerical model of the optical environment from whicha future image will later be recovered as provided for by aspects of theinvention.

FIG. 43a depicts a representation of example image recovery processusing an inverse square-matrix representing an approximate inverse modelas provided for by the invention.

Generalized Inverse/Pseudo-Inverse Remarks

There are a number of types of generalized inverses that have beendeveloped and surveyed in the literature; for example see [B9] Section3.3, [B10] pp. 110-111, and the tables in [B11] pp. 14-17. Some types ofgeneralized inverses are uniquely-defined while others are non-uniquelydefined in terms of infinite families. The notion of a generalizedinverse applies to not only to finite-dimensional matrices but morebroadly to (infinite-dimensional) linear operators; see for example[B12].

The Moore-Penrose generalized inverse, a special case of the Bjerhammar“intrinsic inverses” (see [B10] p. 105 and [P1], is uniquely-defined([B13] p. 180), exists for any rectangular (or square) matrix regardlessof matrix rank ([B13] p. 179, [14]) p. 196, [15] p. 19) and providesmany properties found in matrix inverse ([B14], p. 196) and beyond.

In particular, the Moore-Penrose generalized inverse inherently providesa unique solution providing a “Least-Squares” statistical fit in caseswhere solvable subsets of the larger number of equations give differentinconsistent solutions; see for example [B15] pp. 17-19.

There are other types of generalized inverses that also provideleast-squares properties; for example see entries annotated (3.1) and(3.2) in the table pp. 14 as well as sections 3.1-3.2 of [B11] as wellas section 4.4.1 of [B13].

Further, the Moore-Penrose generalized inverse can be used to determinewhether a solution to a set of linear equations exists ([B13] pp.190-191).

Various extended definitions and generalized forms of the Moore-Penrosegeneralized inverse exist; see for example section 4.4.3 of [B13].

Some of the other types of generalized inverses are not useful forsolving over-specified systems of linear equations (more equations thanvariables), for example the Drazin inverse which is restricted to squarematrices and has more abstract applications; see for example [B13]Section 5.5.

FIG. 43b depicts a representation of example image recovery processusing a generalized-inverse or pseudo-inverse matrix representing anapproximate pseudo-inverse underspecified model as provided for by theinvention. Underspecified arrangements have been of interest insub-Nyquist rate “compressive sampling” which can also be applied morebroadly than optical sensing (see for example [P60]), and naturally fitinto the classical regularized ill-posed inverse problem paradigmemployed endemically in the coded aperture lensless imaging systems andapproaches reviewed in review section A at the opening of the presentpatent application. The regularized ill-posed inverse problem paradigmgives rise to Moore-Penrose pseudo-inverse matrices or matrices similarto those (as presented in [P4] for example). Additionally, it is notedthat the Moore-Penrose pseudo-inverse matrices and some of the othertypes of generalized inverse matrices can provide “best-fit” (Leastsquare error) solutions to underspecified (fewer numbers of measurementsthan needed to uniquely solve for an image), fully-specified (exactlythe number of measurements needed to uniquely solve for an image), andoverspecified (fewer numbers of measurements than needed to uniquelysolve for an image) situations and arrangements. However, the use of theMoore-Penrose pseudo-inverse matrices and other types of generalizedinverses as taught in the inventor's 2008 patent family are directed todeliver additional advantages for pre-designed overspecified measurementsituations and not the result of the regularized ill-posed inverseproblem paradigm now widely used in coded aperture imaging and manyother types of optical systems design and analysis see for example [B5],[B6]).

FIG. 43c depicts a representation of example image recovery processusing a generalized-inverse or pseudo-inverse matrix representing anapproximate generalized-inverse or pseudo-inverse overspecified model asprovided for by the invention.

Use of a generalized-inverse or pseudo-inverse (and use of theMoore-Penrose pseudo-inverse in particular) in solving for a “best-fit”image from overspecified (and likely inconsistent) measurement data wasintroduced in a 2008 inventor's patent family. It is noted thatslightly-related work in the area of improving digital Image resolutionby use of “oversampling” can be found in the far earlier publication byWiman [P3], but that is a different idea and goal. Rather, theinventor's use of use of a generalized-inverse or pseudo-inverse (andMoore-Penrose pseudo-inverse in particular) in solving for a “best-fit”image from overspecified (and likely inconsistent) measurement dataprovides robustness of image recovery with respect to damage oroccultation of portions of the image sensor, etc.

FIG. 44a depicts a simple computational approach for image recovery asprovided for by the invention, for example as taught in the inventor's1999 and 2008 patent families. Such a depicts a simple computationalapproach compares favorably with far more numerically-complex spectralor transform computation (which numerically amounts to additional basisrotation transformation steps) as would be used in spectral or transformmethods. For comparison, FIG. 44b depicts an example representation offar more numerically-complex spectral or transform computation (whichnumerically amounts to additional basis rotation transformation steps)as would be used in spectral or transform methods.

The inventors comprehensive lensless light-field imaging program(beginning with the inventor's 1999 patent family) includes a frameworkcovering the approaches depicted in FIGS. 44a and 44b and varioussituations leading to these such as deconvolution methods. It is theinventor's view that the inventor's comprehensive lensless light-fieldimaging program (beginning with the inventor's 1999 patent family)includes a framework admitting most of visual-light coded aperture andangular-selective light-sensor approaches to various degrees.

FIG. 44c depicts a comparison of the approach depicted in FIG. 44a andthe approach depicted in FIG. 44 b, demonstrating comparative reasons toreject the far more numerically-complex spectral or transformcomputational approached represented in FIG. 44b .

FIG. 45 depicts the use of classical ill-posed inverse-problemregularization methods as employed in the Rambus [P4] and RiceUniversity “Flat Cam” [P5] implementations as well as other codedaperture imaging implementations. These classical ill-posedinverse-problem regularization methods are widely used in many areas butalso have an established role in optical systems design and analysis;see for example [B5] and [B6].

FIG. 46 depicts an abstract representation of an “Identity structure”within a (necessarily-sparse) identity 4-tensor.

FIG. 47 depicts an abstract representation of a dilation around the“Identity structure” within a (necessarily-sparse) Identity 4-tensor.

FIG. 48 depicts a ((7×7)×(7×7)) “matrix-of-matrices” representation of a(7×7×7×7) Identity 4-tensor as abstracted in FIG. 46.

FIG. 49 depicts an example ((7×7)×(7×7)) “matrix-of-matrices”representation of the dilation around the “Identity structure” of a(7×7×7×7) 4-tensor as abstracted in FIG. 46.

FIG. 50 depicts an abstract representation of the trade-off betweenSpace/Inverse-Space/Spatial Frequency Localization Methods and thesparcity of numerical model tensors, matrices, and their inverses.

FIGS. 51a through 51e depict how the sparcity of an example numericalmodel matrix serializing a numerical model 4-tensor degrades as theimage source moves farther and farther from the image sensor (usingnumerical models at each distances predicted by an analytical geometricpredictive model provided for by the invention).

FIG. 52 depicts an example ((7×7)×(7×7)) “matrix-of-matrices”representation, of the numerical model 4-tensor and example, measuredimage (right) at zero separation distance from asingle-illuminated-pixel source image (left). As described elsewhere,the source image need not be on a parallel plane and can be distributedarbitrarily in space.

FIG. 53 depicts an example ((7×7)×(7×7)) “matrix-of-matrices”representation of the numerical model 4-tensor and example measuredimage (right) at a small non-zero separation distance from the samesingle-illuminated-pixel source image (left) as in FIG. 52.

FIG. 54 depicts an example ((7×7)×(7×7)) “matrix-of-matrices”representation of the numerical model 4-tensor and example measuredimage (right) at zero separation distance from a two-illuminated-pixelsource image (left). As described elsewhere, the source image need notbe on a parallel plane and can be distributed arbitrarily in space.

FIG. 55 depicts an example ((7×7)×(7×7)) “matrix-of-matrices”representation of the numerical model 4-tensor and example measuredimage (right) at a small non-zero separation distance from the sametwo-illuminated-pixel source image (left) as in FIG. 54.

FIG. 56 depicts an example ((7×7)×(7×7)) “matrix-of-matrices”representation of the numerical model 4-tensor and example measuredimage (right) zero separation distance from a more closely-spacedtwo-illuminated-pixel source image (left). As described elsewhere, thesource image need not be on a parallel plane and can be distributedarbitrarily in space.

FIG. 57 depicts an example ((7×7)×(7×7)) “matrix-of-matrices”representation of the numerical model 4-tensor and example measuredimage (right) at a small non-zero separation distance from the same moreclosely-spaced two-illuminated-pixel source image (left) as in FIG. 56.

FIGS. 58a through 58c depict three of forty-nine steps of an exampleempirical training sequence as provided for by the invention. In animplementation of the invention, such a procedure could be performed fora selected collection of one or more separation distances between theimage sensor and the source image. As described elsewhere, the sourceimage need not be on a parallel plane and can be distributed arbitrarilyin the observable space.

Lensless Light-Field Imaging as an Associated Inverse Problem

The Inverse Model depiction illustrated in FIG. 1 can be configured to,in some appropriate manner, undo the effects of the incoming light'soptical travel first within the Light-Field preceding the opticalstructure(s) and then through the Lensless Optical Structure(s) to whereit reaches the Optical Sensor where the incident light converted to anelectrical signal that can be further processed. If a mathematical modelis a close match to the composite effects of these optical andoptoelectrical processes, and if the model is mathematically invertible,applying the inverse of the model to the measured data can create acomputationally-produced image which, for example, can be furtherarranged to be useful for human or machine use.

The Inverse Model can, for example, be implemented as a matrix, a4-tensor, or other mathematical and/or data and/or logical operation.

The Inverse Model can be fixed or adjustable, can be implemented in adistributed manner, and can be unique or variationally-replicated invarious manners. The optical structure can be fixed or reconfigurable,and can be arranged to be in a fixed position with respect to theOptical Sensor or can be configured to be movable in some manner withrespect to the Optical Sensor. Additionally, at this level ofabstraction, one or both of the Optical Sensor and Lensless OpticalStructure(s) themselves can be variable in their electrical, physical,optical, mechanical, and other characteristics. For example, one or bothof the Optical Sensor and Lensless Optical Structure(s) themselves canbe any one or more of flat, curved, bendable, elastic,elastically-deformable, plastically-deformable, etc.

The Inverse Model can be derived from analytical optical models,empirical measurements, or combinations of these. In some embodimentsthe Inverse Model can be parametrized using interpolation.

Interpolation-based parameterization can be particularly useful if theInverse Model is based on a collection of selected empiricalmeasurements, or if the analytical optical model involves complexnumerical computations.

FIG. 59 depicts an example polynomial fitting function interpolationmethod for interpolating the numerical model, itsinverse/pseudo-inverse, and/or other functions for separation distancesvalues lying between k empirically-trained separation distances. Asdescribed elsewhere, the source image need not be on a parallel planeand can be distributed arbitrarily in the observable space. In generalthe polynomial fitting function can be expected to include terms withnegative exponential powers expected due to the overall “1/r²”enveloping attention as the serration distance “r” increases.

FIG. 60 depicts an example polynomial fitting function interpolationmethod for interpolating the numerical model, itsinverse/pseudo-inverse, and/or other functions for separation distancesvalues lying between k separation distances used by a predictive-model.As described elsewhere, the source image need not be on a parallel planeand can be distributed arbitrarily in the observable space. Hereto ingeneral the polynomial fitting function can be expected to include termswith negative exponential powers expected due to the overall envelopingattention as the serration distance “r” increases.

FIG. 61 depicts an example of very poor curve-fit interpolation of thematrix elements of the numerical model between measure data distancesresulting from not having sufficient terms in a model polynomialexpansion and/or inclusion of terms with negative exponential powersexpected due to the overall “1/r²” enveloping attention as the serrationdistance “r” increases.

FIG. 62 depicts an example piecewise-linear interpolation of the matrixelements of the numerical model between k empirically-trained separationdistances. (Measurement data and piecewise-linear plot made by MichaelHörlng.)

Using an empirically-trained numerical model for representing the lineartransformation invoked by the optical arrangement, it is clearlypossible to train the system to focus on an arbitrarily-shaped surface,including one that is curved, bend, or irregularly-shaped; the inversionmath “does not care” as long as the resulting numerical model matrix isnon-singular, and the recovered image will be obtained in the samemanner as if the focus-surface was a parallel plane. Accordingly, inprinciple a predictive analytical model can be used to generate thenumerical model matrix, and by either means (empirically-trained orpredictively-modeled) the system and methods can be arranged to focus onan arbitrarily-shaped surface, including one that is curved, bend, orirregularly-shaped.

FIG. 63a depicts how with very small separation distances and certainnon-alignment of source-pixel locations and sensor pixel locations somesensor pixel locations cannot receive light from proximate source-pixellocations, while FIG. 63b depicts how at slightly greater separationdistances this condition does not occur; such processes can give rise tothe rising and falling of selected curves in the example empiricaltraining data shown in the example of FIG. 61. (This analysis performedby Michael Hörlng.)

FIG. 64 depicts an example piecewise-linear interpolation method forinterpolating the numerical model, its inverse/pseudo-inverse, and/orother functions for separation distances values lying between kempirically-trained separation distances. As described elsewhere, thesource image need not be on a parallel plane and can be distributedarbitrarily in the observable space.

FIG. 65 depicts an example piecewise-linear interpolation method forinterpolating the numerical model, its inverse/pseudo-inverse, and/orother functions for separation distances values lying between kseparation distances used by a predictive-model. As described elsewhere,the source image need not be on a parallel plane and can be distributedarbitrarily in the observable space.

FIG. 66 depicts an abstract representation of the mathematical'recoveryof image from measured image array data.

FIG. 67 depicts a variation on the representation of the mathematicalrecovery of image from measured image array data shown in FIG. 66wherein more measurements are made than needed, resulting in anover-specified collection of measurements that can be expected to leadto inconsistent calculation outcomes when different subgroups ofmeasurements are used to solve for the recovered image. As provided forby the invention, a Moore-Penrose pseudo-inverse operation gives aleast-squares fit to the expected inconsistent outcomes; this isdepicted in the bottom portion of the figure.

FIG. 68a through 68d depict example numerical model outcomes responsiveto a single-illuminated pixel as generated for various separationdistances by an example predictive analytical geometric model. Theparticular example predictive analytical geometric model used onlyaccounts for vignette occultation as calculated by simple ray-trancingand does not include the effects of vignette-internal reflections,vignette-internal scattering, vignette-aperture diffraction, surfaceplasmoid propagation, etc.

FIG. 69a depicts three example interactive quantization-effect modelingoutcomes wherein controllable quantization is artificially imposed onmeasurement data so as to predictively model and characterize theeffects of quantizing nonlinearities imposed by electronicdigital-to-analog converter processes on empirical training,predictive-model generated, and real-time measurements as they influencethe quality of image recovery.

FIG. 69b depicts three example interactive offset-effect modelingoutcomes wherein controllable measurement offset is artificially imposedon measurement data so as to predictively model and characterize theeffects of measurement offsets imposed by electronic digital-to-analogconverter processes on empirical training, predictive-model generated,and real-time measurements as they influence the quality of imagerecovery.

There are many noise processes inherent to light sensing and associatedelectronics and various resulting performance limitations and tradeoffs;see for example [P44], [B2]. A very general performance perspective isprovided in the book by Janesick [B2]. In the limit, highest performancewill be obtained by single-electron sensors and amplifies; as to stepstowards array sensors of this type see the paper by Richardson [P33].The invention provides for inclusion of these considerations. FIG. 69cdepicts an example interactive noise-modeling control used to introducesynthetically-generated noise and noise processes so as to predictivelymodel and characterize its effects. The selection shown controlsadditive Gaussian noise, but other noise processes associated withphotodiodes (1/f noise, dark-current shot (Poissonian) noise, photonshot (Poissonian) noise, Johnson and other circuit noise, dark-currentthermal noise, spectral noise, detector amplifier noise, ect.) cansimilarly be introduced.

FIGS. 70a through 70d depict example interactive modeling outcomesshowing the effect of noise, offset, and both these for a 32-step(5-bit) quantized (“DN” [B2]) measurement dynamic range.

FIGS. 71a through 71d depict example interactive modeling outcomesshowing the effect of noise, offset, and both these for a 64-step(6-bit) quantized (“DN”) measurement dynamic range.

FIGS. 72a through 72d depict example interactive modeling outcomesshowing the effect of noise, offset, and both these for a 128-step(7-bit) quantized (“DN”) measurement dynamic range.

FIGS. 73a through 73d depict example interactive modeling outcomesshowing'the effect of noise, offset, and both these for a 140-step(slightly more than 7-bit) quantized (“DN”) measurement dynamic range.

FIGS. 74a through 74d depict example interactive modeling outcomesshowing the effect of noise, offset, and both these for a 150-step (yetmore than 7-bit) quantized (“DN”) measurement dynamic range.

FIG. 75 a, adapted from [B7] Figure C2.5.16(a), depicts a first examplepixel-cell multiplexed-addressing circuit for an individual OLED withinan OLED array that includes a dedicated light-coupled monitoringphotodiode for use in regulating the light output of the individual OLEDso as to prevent user-observed fading or other brightness-variationprocesses. The adjacent photodiodes used for pixel-by-pixel closed-loopfeedback of OLED brightness.

FIG. 75 b, adapted from [B7] Figure C2.5.16(b), depicts a second examplepixel-cell multiplexed-addressing circuit for an individual OLED withinan OLED array that includes a dedicated light-coupled monitoringphotodiode for use in regulating the light output of the individual OLEDso as to prevent user-observed fading or other brightness-variationprocesses. There are many other subsequent developments since thepublishing of this books tentatively-toned remarks; for examplerecently-announced OLED phones are said to be using this technique.

FIG. 76 a, adapted from [B18], depicts an example pixel-cellmultiplexed-addressing circuit for an individual OLED within an OLEDarray with a monitoring feature.

FIG. 76 b, also adapted from [B18], depicts an example pixel-cellmultiplexed-addressing circuit for an isolated high-performancephotodiode or phototransistor within a high-performance photodiode orphototransistor array with a forced-measurement provision.

Additional Functional Architectures

As described earlier, FIG. 1 depicts an, example conceptual view of theunderlying principles of the invention, facilitating a wide range ofimplementation methods and architectures. In this depiction, an OpticalScene creates a Light-Field that is directed to an Optical Sensor whichis preceded by one or more Lensless Optical Structure(s) that in somemanner alters the light field in a predictable spatial manner. TheOptical Sensor produces (typically time-varying) electrical signalsand/or computational data responsive (instantly and/or within sometime-delay) to light incident to the surface or other substructure(s)within the Optical Sensor at any given moment. The depicted InverseModel can be configured to, in some appropriate manner, undo the effectsof the incoming light's optical travel first within the Light-Fieldpreceding the optical structure(s) and then through the Lensless OpticalStructure(s) to where it reaches the Optical Sensor, resulting in acomputationally-produced image which, for example, can be arranged to beuseful for human or machine use. The Inverse Model can, for example, beimplemented as a matrix, a 4-tensor, or other mathematical and/or dataand/or logical operation. The Inverse Model can be fixed or adjustable,can be implemented in a lumped or distributed manner, and can be uniqueor variationally-replicated in various manners. The optical structurecan be fixed or reconfigurable, and can be arranged to be in a fixedposition with respect to the Optical Sensor or can be configured to bemovable in some manner with respect to the Optical Sensor. Additionally,at this level of abstraction, one or both of the Optical Sensor andLensless Optical Structure(s) themselves can be variable in theirelectrical, physical, optical, mechanical, and other characteristics.For example, one or both of the Optical Sensor and Lensless OpticalStructure(s) themselves can be any one or more of flat, curved,bendable, elastic, elastically-deformable, plastically-deformable, etc.

FIG. 77 depicts a simplified view of FIG. 1 showing only the signal andcomputational portion of FIG. 1 as will be useful in a subsequentdiscussion. It is understood that in FIG. 77 and related figures thatthe Optical Scene, Light-Field, and Lensless Optical Structure(s) canhave the arrangements with respect to the Optical Sensor like that,similar to, extensible from, or in appropriate manners alternative tothat depicted in FIG. 1.

FIG. 78 depicts a variation of the arrangement represented in FIG. 77wherein the Inverse Model is rendered as a parameterized Inverse Modelwhich can be altered responsive to one or more provided parameters. Forexample, the parameters provided to the parameterized Inverse Modelcould control a surrogate viewpoint, specify one or more “focus planes”(or more generally “focus surfaces”), etc. rating to what the InverseModel “undoes.” Accordingly, this arrangement allows a plurality ofimaging capabilities and functions to be selectably and or adjustably besupported by the more general arrangement of FIG. 77.

FIG. 79 depicts a variation of the arrangement represented in FIG. 78wherein the electrical signals and/or computational data produced by theOptical Sensor are in parallel provided in whole or selected (orselectable) part to a plurality of Inverse Models, each producing one ormore computationally-produced images responsive to the sensor data.Although the Inverse Models shown in FIG. 79 are depicted asparameterized Inverse Models, the invention provides for some or all ofthe plurality of Inverse Models to be non-parameterized Inverse Models.

FIG. 80 depicts a variation of the arrangement represented in FIG. 78wherein the electrical signals and/or computational data produced by theOptical Sensor is handled by a computer or computational element (suchas a microprocessor, GPU, DSP chip, ALU, FPLA, combination of two ormore these, pluralities of these, etc.) in some fashion that at leastpermits the electrical signals and/or computational data produced by theOptical Sensor to be stored as a file.

FIG. 81 depicts a variation of the arrangement represented in FIG. 78wherein the aforementioned handling by a computer or computationalelement is controlled in some manner by a control parameter. The controlparameter for example can specify an aspect of the name of the StoredFile, specify an aspect of the format of the Stored File, specify aselection of specific portions of the electrical signals and/orcomputational data produced by the Optical Sensor, specify or controldata operations on the electrical signals and/or computational dataproduced by the Optical Sensor, specify or mathematical operations onthe electrical signals and/or computational data produced by the OpticalSensor, specify or control logical operations on the electrical signalsand/or computational data produced, by the Optical Sensor, etc.

FIG. 82a depicts a variation of the arrangement represented in FIG. 81wherein a plurality of stored files is created, for example withdifferent parameter values associated with each stored file. Although aseparate computer function is depicted for each of the stored file, theinvention provides for these computer functions to be implemented withor executed on any of a single computer or computational element (suchas a microprocessor, GPU, DSP chip, ALU, FPLA combination of two or morethese, pluralities of these, ect.), a plurality of computers orcomputational elements, an individually-dedicated computer orcomputational element, etc. The parameter values used in creating eachStored File can be either externally associated with each stored file orcan be stored as part of the stored file in a direct or encoded form.

FIG. 82b depicts an example arrangement wherein a stored file is used bya fixed Inverse Model to create a computationally-produced image.

FIG. 82c depicts an example arrangement wherein a stored file is used bya parameterized Inverse Model to create a computationally-producedimage, and further wherein parameter value(s) associated theparameterized Inverse Model are externally associated with each storedfile.

FIG. 82d depicts an example arrangement wherein a stored file is used bya parameterized Inverse Model to create a computationally-producedimage, and further wherein parameter value(s) associated theparameterized Inverse Model are derived from or obtained from the storedfile.

FIG. 83 depicts four example conformations of a particular bendable,flexible, and/or or pliable optical-sensor/optical-structure sheet, eachconformation giving rise to an associated model for how a light field issensed in the context of reconstructing an image, and each model givingrise to its own associated inverse model. It is noted that, as dependenton the properties and limitations of the optical-structure, there can besmall blind spots in regions of sufficiently-high curvature.

FIG. 84a through FIG. 84d depict how various conformations can be usedto render a computationally derived image.

FIG. 85a depicts an example arrangement for anoptical-senor/optical-structure of fixed conformation.

FIG. 85b depicts an example arrangement for anoptical-senor/optical-structure of variable conformation for exampleshould the surface bend, deform, hinge, expand, contract, etc.

FIG. 85c depicts a variation of FIG. 85b wherein the model and theInverse Model are parameterized.

FIG. 86a depicts an arrangement useful for using optical training tosense the present conformation of an optical-senor/optical-structure ofvariable conformation, for example should the surface bend, deform,hinge, expand, contract, etc.

FIG. 86b depicts an arrangement useful for using internal sensing meansto sense the present conformation of an optical-senor/optical-structureof variable conformation, for example should the surface bend, deform,hinge, expand, contract, etc. It is noted that a prototype for acommercial (panoramic camera) product employing a flexible OLED display,(c) conformation deformation sensing has been presented and is describedin [P36].

FIG. 86c depicts an arrangement useful for using information about amovable or changing support structure or contact arrangement to identifythe present conformation of an optical-senor/optical-structure ofvariable conformation, for example should the surface bend, deform,hinge, expand, contract, etc.

FIG. 86d depicts an arrangement useful for using external observationmean, for example such as one or more observing video camera(s) means tosense the present conformation of an optical-senor/optical-structure ofvariable conformation, for example should the surface bend, deform,hinge, expand, contract, etc.

FIG. 87a depicts an optical-senor/optical-structure of variableconformation, for example should the surface bend, deform, hinge,expand, contract, etc. producing an output signal and/or output data.

FIG. 87b depicts a controllable variable-conformation material whoseshape/conformation can be controlled by externally-provide controlstimulus.

FIG. 87c depicts an optical-senor/optical-structure of variableconformation, for example should the surface bend, deform, hinge,expand, contract, etc, producing an output signal and/or output datafabricated on, with, or co-integrated with controllablevariable-conformation material whose shape/conformation can becontrolled by externally-provide control stimulus.

FIG. 88a depicts use of conformational sensing information to derive orcompute parameter values for a parameterized Inverse Model.

FIG. 88b depicts use of conformational control parameter information toderive or compute parameter values for a parameterized Inverse Model.

FIG. 88c depicts use of both conformational sensing information andconformational control parameter information to derive or computeparameter values for a parameterized Inverse Model.

Imaging Algorithms

The development to follow is broad enough to cover a wide variety ofsensor types and imaging frameworks, and can be expanded further.Although the development to follow readily supports the advancedfeatures made possible by curved, flexible/bendable, transparent,light-emitting, and other types of advanced sensors taught in thepresent patent application and earlier inventor patent families, many ofthe techniques can be readily applied to appreciate opticalarrangements, devices, and situations employing traditional imagesensors such as CMOS, CCD, vidicon, etc. Accordingly, the presentinvention provides for the use of a wide range of image sensor typesincluding CMOS, COD, vidicon, flat, curved, flexible/bendable,transparent, light-emitting, and other types of advanced sensors taughtin the present patent application and earlier inventor patent families,as well as other know and future types of image sensors.

Traditional Notation Conventions for Vectors, Matrices, andMatrix-Vector (Left) Multiplication

Let x be an M-dimensional column vector (i.e., an array of dimension1×M) comprising elements {x_(m)} 1≤m≤M

$x = \begin{bmatrix}x_{1} \\\vdots \\x_{M}\end{bmatrix}$and y be a j-dimensional column vector (i.e., an array of dimension 1×M)comprising elements {y_(j)} 1≤j≤J

$y = \begin{bmatrix}y_{1} \\\vdots \\y_{J}\end{bmatrix}$

Let A be a J×M matrix (an array dimension j×M) comprising elements{a_(jm)}, 1≤j≤J, 1≤m≤M:

$A = \begin{bmatrix}a_{11} & \ldots & a_{1K} \\\vdots & \ddots & \vdots \\a_{J\; 1} & \ldots & a_{JK}\end{bmatrix}$

The “matrix product” of a J×M matrix A with an M-dimensional columnvector x can produce a J-dimensional column vector y; by “leftmultiplication” convention this is denoted asy=Axwhere each element y_(j) of resulting a J-dimensional column vector y iscalculated as

$y_{j} = {\sum\limits_{m = 1}^{M}{a_{jm}x_{m}}}$for each j an integer such that 1≤j≤J. Then the entire J-dimensionalcolumn vector y is given by

$y = {\begin{bmatrix}y_{1} \\\vdots \\y_{J}\end{bmatrix} = \begin{bmatrix}{\sum\limits_{m = 1}^{M}{a_{1m}x_{m}}} \\\vdots \\{\sum\limits_{m = 1}^{M}{a_{Jm}x_{m}}}\end{bmatrix}}$Use to Represent Spatial Line-Array (1-Dimensional) Imaging (as used inFax and Spectrometry Sensors)

In the above, one is using the matrix A as a transformational mappingfrom column vector x to column vector y

$x\overset{A}{\rightarrow}y$analogous 4-Tensor Representation of Spatial Grid-Array (2-dimensional)Imaging.

For example, if column vector x represents a line-array of data (such aslight source values directed to a line-array light-measurement sensorused in a fax scanner or optical spectrometer), the matrix A canrepresent a composite chain of linear optical processes, electro-opticalprocesses, and interface electronics (transconductance, transimpedance,amplification, etc.) processes that result in measured data representedby a column vector y. Here the indices of the vectors and matrix signifyunique well-defined discrete (step-wise) spatial positions in anunderlying 1-dimensional spatial structure.

A visual image as humans experience it through vision and conventionalphotography, as well as other analogous phenomena, has an underlying2-dimensional spatial structure. In digital imaging, unique well-defineddiscrete (step-wise) spatial positions within an underlying2-dimensional spatial structure are identified and/or employed—in thecontext of this discussion these may be called “pixels.”

Mathematically, a 2-dimensional array of mathematically-valued elements,such as a matrix, can provide a mathematical representation of an imagewherein the indices of an element identify an individual pixel's spatiallocation and the value of that mathematical element represents the“brightness” of that pixel. For example, an J×K array of measured2-dimensional image data arranged a row and columns can be presented asa matrix of “brightness” values:

$Q = \begin{bmatrix}q_{11} & \ldots & q_{1K} \\\vdots & \; & \vdots \\q_{J\; 1} & \ldots & q_{JK}\end{bmatrix}$

A convenient shorthand for this can be denoted asQ={q_(jk)}where understood each of the two indices span a range of consecutivenon-zero integer values1≤j≤J, 1≤k≤K.

Similarly, a source image (2-dimensional array) can be represented as amatrix:

$S = \begin{bmatrix}s_{11} & \ldots & s_{1N} \\\vdots & \; & \vdots \\s_{M\; 1} & \ldots & s_{MN}\end{bmatrix}$

A similar convenient shorthand for this is denotedS={S_(mn})where it is understood each of the two induces span a range ofconsecutive non-zero integer values1≤m≤M, 1≤n≤N.

A source image S can be transformed by linear optical processes linearsensor processes, and linear electronics processes into a measured imageQ. This can be represented mathematically as a linear matrix-to-matrixtransformation

mapping the matrix S to the matrix Q

$S\overset{\mathbb{T}}{\rightarrow}Q$akin to employing the earlier matrix A as a linear vector-to-vectortransformational (for example, mapping column vector x to column vectory):

$x\overset{A}{\rightarrow}y$

Most generally this linear transformation

can be represented by a 4-dimensional array 4-tensor:

={t_(jkmn})1≤j≤J1≤k≤K1≤m≤M1≤n≤Nwith the understanding that the following multiplication rule isrepresented by the tensor

“multiplying” the matrix S, namely each element q_(jk) of resultingmatrix Q is given by

$q_{jk} = {\sum\limits_{m = 1}^{M}{\sum\limits_{n = 1}^{N}\;{t_{jkmn}s_{mn}}}}$1 ≤ j ≤ J, 1 ≤ k ≤ K.

Note this convention corresponds in form to the matrix case presentedearlier:

$y_{j} = {\sum\limits_{k = 1}^{K}{a_{jk}x_{k}}}$ 1 ≤ k ≤ K

The corresponding “multiplicative” product of the 4-tensor

with matrix S to give matrix Q can be represented asQ=

Swhich compares analogously to the “multiplicative” product of the matrixA with the vector x to give the vector yy=Ax

With its four tensor-element indices and tensor-matrix product organizedin this way, the 4-tensor

with elements t_(jkmn) can be readily and conveniently represented as amatrix-of-matrices where interior matrix blocks are indexed by row j andcolumn k and the elements with each of the interior matrices are indexedby row m and column n. More specifically, the 4-tensor element t_(jkmn)resides in an inner matrix residing in row m and column n of an innermatrix that resides in row j and column k of the outer matrix. Theresulting matrix-of-a-matrices representation for the 4-tensor

={t_(jkmn)} with1≤j≤J, 1≤k≤K, 1≤m≤M, 1≤n≤Nwould be:

${\mathbb{T}} = \begin{bmatrix}{\begin{bmatrix}t_{1111} & \ldots & t_{111N} \\\vdots & \ddots & \vdots \\t_{11M\; 1} & \ldots & t_{11{MN}}\end{bmatrix}\mspace{14mu}{\ldots\mspace{11mu}\begin{bmatrix}t_{1K\; 11} & \ldots & 1_{1K\; 1N} \\\vdots & \ddots & \vdots \\t_{1{KM}\; 1} & \ldots & t_{1{KMN}}\end{bmatrix}}} \\\vdots \\{\begin{bmatrix}t_{J\; 111} & \ldots & 1_{J\; 11N} \\\vdots & \ddots & \vdots \\t_{J\; 1M\; 1} & \ldots & t_{J\; 1{MN}}\end{bmatrix}\mspace{14mu}{\ldots\mspace{11mu}\begin{bmatrix}t_{{JK}\; 11} & \ldots & 1_{{JK}\; 1N} \\\vdots & \ddots & \vdots \\t_{{JKM}\; 1} & \ldots & t_{JKMN}\end{bmatrix}}}\end{bmatrix}$

This matrix-of-a-matrices structure where the mapping

$S\overset{\mathbb{T}}{\rightarrow}Q$is defined by

$q_{jk} = {\sum\limits_{m = 1}^{M}{\sum\limits_{n = 1}^{N}\;{t_{jkmn}s_{mn}}}}$1 ≤ j ≤ J, 1 ≤ k ≤ K.provides several important opportune outcomes, among these being:

-   -   Property: The individual entries t_(jkmn) of each interior        matrix having block-position index {j, k} scale the contribution        of each element S_(mn) which sum together to the quantity q_(jk)        comprised within the matrix Q; this can be seen directly by just        considering fixed values for {j, k} in the defining relation

$q_{jk} = {\sum\limits_{m = 1}^{M}{\sum\limits_{n = 1}^{N}\;{t_{jkmn}s_{mn}}}}$

-   -   Utility: this is also the way MatrixForm[*] displays the 4-array        in Mathematica.™

Remark 1: It is noted that the 4-tensor

as defined thus far is represented in this “matrix of matrices”structure, the interior matrices are organized so that each interiormatrix having block-position index {j, k} is associated with thecorresponding outcome quantity q_(jk). An attractive property of thisrepresentation, as called out above, is that the value of the outputquantity q_(jk) is the sum of all the pointwise products of values ofsource image pixels s_(mn) scaled by the corresponding elements in theinterior matrix that has block-position index {j, k}. Thus, in anoptical imaging context, the values of elements in the interior matrixthat has block-position index {j, k} graphically show the multiplicative“gain” (or “sensitivity”) attributed to each of the same-positionedsource image pixels s_(mn) in the image source matrix S. In morepedestrian but intuitively useful terms, the values of elements in theinterior matrix that has block-position index {j, k} display the “heatmap” of responsiveness of an observed or measured sensor pixel q_(jk) inthe observed or measured image matrix Q to source image pixels s_(mn) inthe image source matrix S.

Remark 2 From this it i further noted that other kinds of 4-tensorscould be reorganized in other ways that have other attractive merits.For example, a 4-tensor Ψ comprising elements Ψ_(mnjk) can be defined bythe simple index reorderingΨ_(mnjk)=t_(jkmn);each interior matrix in the “matrix of matrices” structure for the4-tensor Ψ having block-position index {m, n} represents a discrete“point-spread function” for a source pixel at position {m, n} intoindividual outcome pixels at position {j, k} as can be seen from theresulting relation

$q_{jk} = {\sum\limits_{m = 1}^{M}{\sum\limits_{n = 1}^{N}{\psi_{mnjk}\; s_{mn}}}}$

Although “point-spread function” representation imposed by the “matrixof matrices ” structure for 4-tensor Ψ has obvious customary attraction,the discourse will continue in terms of the 4-tensor

comprising elements t_(jkmn) as defined by

$q_{jk} = {\sum\limits_{m = 1}^{M}{\sum\limits_{n = 1}^{N}\;{t_{jkmn}s_{mn}}}}$because of its organizational similarity with the conventional matrixdefinition

$y_{j} = {\sum\limits_{k = 1}^{K}{a_{jk}x_{k}}}$and with the understanding that all the subsequent development can betransformed from the definitions used for 4-tensor

to the “point-spread” oriented for 4-tensor Ψ by the index re-mappingΨ_(mnjk)=t_(jkmn).

Remark 3: It is noted that for a (variables-separable) “separable”two-dimensional transform, such as the two-dimensional DFT, DCT, DST,etc., commonly used in traditional spectral image processing affairs ofthe j and m indices are handled entirely separate from affairs of the kand n indices, so q_(jk) takes the restricted “variables-separable”form, for example when J=M and K=N

$q_{ik} = {\sum\limits_{m = 1}^{M}{\sum\limits_{n = 1}^{N}{d_{jm}d_{kn}s_{mn}}}}$in which caset_(jkmn)=d_(jm)d_(kn)

For example, for the normalized DFT matrices operating on an image of Mrows and N columns, these d_(jm) and d_(kn) element are:

$d_{jm} = \frac{e^{{- 2}\pi\;{i{({j - 1})}}{({m - 1})}}}{\sqrt{M}}$ and$d_{kn} = \frac{e^{{- 2}\pi\;{i{({k - 1})}}{({n - 1})}}}{\sqrt{N}}$where i=√{square root over (−1)} .

As another example of other kinds of 4-tensors be reorganized in otherways with attractive merits, a variables-separable 4-tensor Φ comprisingelements ϕ_(mnjk) can be defined by the simple index reorderingϕ_(jmkn)=t_(jkmn)which separately associates rows (indexed by j) in matrix Q with rows(indexed by m) in matrix S and separately associates columns (indexed byk) in matrix Q with columns (indexed by n) in matrix S.

Remark 4: To further illustrate details and develop intuition of the“Matrix of matrices” structure, or at least the aforedescribedorganization of indices and multiplication rule, one could employ thedimension signifier (J×K)×(M×N). As some examples:

$\begin{matrix}{{\mathbb{T}}_{{({3 \times 3})} \times {({3 \times 3})}} = \begin{bmatrix}\begin{bmatrix}t_{1111} & t_{1112} & t_{1113} \\t_{1121} & t_{1122} & t_{1123} \\t_{1131} & t_{1132} & t_{1133}\end{bmatrix} & \begin{bmatrix}t_{1211} & t_{1212} & t_{1213} \\t_{1221} & t_{1222} & t_{1223} \\t_{1231} & t_{1232} & t_{1233}\end{bmatrix} & \begin{bmatrix}t_{1311} & t_{1312} & t_{1313} \\t_{1321} & t_{1322} & t_{1323} \\t_{1331} & t_{1332} & t_{1333}\end{bmatrix} \\\begin{bmatrix}t_{2111} & t_{2112} & t_{2113} \\t_{2121} & t_{2122} & t_{2123} \\t_{2131} & t_{2132} & t_{2133}\end{bmatrix} & \begin{bmatrix}t_{2211} & t_{2212} & t_{2213} \\t_{2221} & t_{2222} & t_{2223} \\t_{2231} & t_{2232} & t_{2233}\end{bmatrix} & \begin{bmatrix}t_{2311} & t_{2312} & t_{2313} \\t_{2321} & t_{2322} & t_{2323} \\t_{2331} & t_{2332} & t_{2333}\end{bmatrix} \\\begin{bmatrix}t_{3111} & t_{3112} & t_{3113} \\t_{3121} & t_{3122} & t_{3123} \\t_{3131} & t_{3132} & t_{3133}\end{bmatrix} & \begin{bmatrix}t_{3211} & t_{3212} & t_{3213} \\t_{3221} & t_{3222} & t_{3223} \\t_{3231} & t_{3232} & t_{3233}\end{bmatrix} & \begin{bmatrix}t_{3311} & t_{3312} & t_{3313} \\t_{3321} & t_{3322} & t_{3323} \\t_{3331} & t_{3332} & t_{3333}\end{bmatrix}\end{bmatrix}} & \; \\{{\mathbb{T}}_{{({5 \times 4})} \times {({3 \times 2})}} = \begin{bmatrix}\begin{bmatrix}t_{1111} & t_{1112} \\t_{1121} & t_{1122} \\t_{1131} & t_{1132}\end{bmatrix} & \begin{bmatrix}t_{1211} & t_{1212} \\t_{1221} & t_{1222} \\t_{1231} & t_{1232}\end{bmatrix} & \begin{bmatrix}t_{1311} & t_{1312} \\t_{1321} & t_{1322} \\t_{1331} & t_{1332}\end{bmatrix} & \begin{bmatrix}t_{1411} & t_{1412} \\t_{1421} & t_{2422} \\t_{1431} & t_{1432}\end{bmatrix} \\\begin{bmatrix}t_{2111} & t_{2112} \\t_{2121} & t_{2122} \\t_{2131} & t_{2132}\end{bmatrix} & \begin{bmatrix}t_{2211} & t_{2212} \\t_{2221} & t_{2222} \\t_{2231} & t_{2232}\end{bmatrix} & \begin{bmatrix}t_{2311} & t_{2312} \\t_{2321} & t_{2322} \\t_{2331} & t_{2332}\end{bmatrix} & \begin{bmatrix}t_{2411} & t_{2412} \\t_{2421} & t_{2422} \\t_{2431} & t_{2432}\end{bmatrix} \\\begin{bmatrix}t_{3111} & t_{3112} \\t_{3121} & t_{3122} \\t_{3131} & t_{3132}\end{bmatrix} & \begin{bmatrix}t_{3211} & t_{3212} \\t_{3221} & t_{3222} \\t_{3231} & t_{3232}\end{bmatrix} & \begin{bmatrix}t_{3311} & t_{3312} \\t_{3321} & t_{3322} \\t_{3331} & t_{3332}\end{bmatrix} & \begin{bmatrix}t_{3411} & t_{3412} \\t_{3421} & t_{3422} \\t_{3431} & t_{3432}\end{bmatrix} \\\begin{bmatrix}t_{4111} & t_{4112} \\t_{4121} & t_{4122} \\t_{4131} & t_{4132}\end{bmatrix} & \begin{bmatrix}t_{4211} & t_{4212} \\t_{4221} & t_{4222} \\t_{4231} & t_{4232}\end{bmatrix} & \begin{bmatrix}t_{4311} & t_{4312} \\t_{4321} & t_{4322} \\t_{4331} & t_{4332}\end{bmatrix} & \begin{bmatrix}t_{4411} & t_{4412} \\t_{4421} & t_{4422} \\t_{4431} & t_{4432}\end{bmatrix} \\\begin{bmatrix}t_{5111} & t_{5112} \\t_{5121} & t_{5122} \\t_{5131} & t_{5132}\end{bmatrix} & \begin{bmatrix}t_{5211} & t_{5212} \\t_{5221} & t_{5222} \\t_{5231} & t_{5232}\end{bmatrix} & \begin{bmatrix}t_{5311} & t_{5312} \\t_{5312} & t_{5322} \\t_{5331} & t_{5332}\end{bmatrix} & \begin{bmatrix}t_{5411} & t_{5412} \\t_{5421} & t_{5422} \\t_{5431} & t_{5432}\end{bmatrix}\end{bmatrix}} & \; \\{{\mathbb{T}}_{{({2 \times 3})} \times {({5 \times 4})}} = \begin{bmatrix}\begin{bmatrix}t_{1111} & t_{1112} & t_{1113} & t_{1114} \\t_{1121} & t_{1122} & t_{1123} & t_{1124} \\t_{1131} & t_{1132} & t_{1133} & t_{1134} \\t_{1141} & t_{1142} & t_{1143} & t_{1144} \\t_{1151} & t_{1152} & t_{1153} & t_{1154}\end{bmatrix} & \begin{bmatrix}t_{1211} & t_{1212} & t_{1213} & t_{1214} \\t_{1221} & t_{1222} & t_{1223} & t_{1224} \\t_{1231} & t_{1232} & t_{1233} & t_{1234} \\t_{1241} & t_{1242} & t_{1243} & t_{1244} \\t_{1251} & t_{1252} & t_{1253} & t_{1254}\end{bmatrix} & \begin{bmatrix}t_{1311} & t_{1312} & t_{1313} & t_{1314} \\t_{1321} & t_{1322} & t_{1323} & t_{1324} \\t_{1331} & t_{1332} & t_{1333} & t_{1334} \\t_{1341} & t_{1342} & t_{1343} & t_{1344} \\t_{1351} & t_{1352} & t_{1353} & t_{1354}\end{bmatrix} \\\begin{bmatrix}t_{2111} & t_{2112} & t_{2113} & t_{2114} \\t_{2121} & t_{2122} & t_{2123} & t_{2124} \\t_{2131} & t_{2132} & t_{2133} & t_{2134} \\t_{2141} & t_{2142} & t_{2143} & t_{2144} \\t_{2151} & t_{2152} & t_{2153} & t_{2154}\end{bmatrix} & \begin{bmatrix}t_{2211} & t_{2212} & t_{2213} & t_{2214} \\t_{2221} & t_{2222} & t_{2223} & t_{2224} \\t_{2231} & t_{2232} & t_{2233} & t_{2234} \\t_{2241} & t_{2242} & t_{2243} & t_{2244} \\t_{2251} & t_{2252} & t_{2253} & t_{2254}\end{bmatrix} & \begin{bmatrix}t_{2311} & t_{2312} & t_{2313} & t_{2314} \\t_{2321} & t_{2322} & t_{2323} & t_{2324} \\t_{2331} & t_{2332} & t_{2333} & t_{2334} \\t_{2341} & t_{2342} & t_{2343} & t_{2344} \\t_{2351} & t_{2352} & t_{2353} & t_{2354}\end{bmatrix}\end{bmatrix}} & \; \\{{\mathbb{T}}_{{({5 \times 4})} \times {({2 \times 3})}} = \begin{bmatrix}\begin{bmatrix}t_{1111} & t_{1112} & t_{1113} \\t_{1121} & t_{1122} & t_{1123}\end{bmatrix} & \begin{bmatrix}t_{1211} & t_{1212} & t_{1213} \\t_{1221} & t_{1222} & t_{1223}\end{bmatrix} & \begin{bmatrix}t_{1311} & t_{1312} & t_{1313} \\t_{1321} & t_{1322} & t_{1323}\end{bmatrix} & \begin{bmatrix}t_{1411} & t_{1412} & t_{1413} \\t_{1421} & t_{1422} & t_{1423}\end{bmatrix} \\\begin{bmatrix}t_{2111} & t_{2112} & t_{2113} \\t_{2121} & t_{2122} & t_{2123}\end{bmatrix} & \begin{bmatrix}t_{2211} & t_{2212} & t_{2213} \\t_{2221} & t_{2222} & t_{2223}\end{bmatrix} & \begin{bmatrix}t_{2311} & t_{2312} & t_{2313} \\t_{2321} & t_{2322} & t_{2323}\end{bmatrix} & \begin{bmatrix}t_{2411} & t_{2412} & t_{2413} \\t_{2421} & t_{2422} & t_{2423}\end{bmatrix} \\\begin{bmatrix}t_{3111} & t_{3112} & t_{3113} \\t_{3121} & t_{3122} & t_{3123}\end{bmatrix} & \begin{bmatrix}t_{3211} & t_{3212} & t_{3213} \\t_{3221} & t_{3222} & t_{3223}\end{bmatrix} & \begin{bmatrix}t_{3311} & t_{3312} & t_{3313} \\t_{3321} & t_{3322} & t_{3323}\end{bmatrix} & \begin{bmatrix}t_{3411} & t_{3412} & t_{3413} \\t_{3421} & t_{3422} & t_{3423}\end{bmatrix} \\\begin{bmatrix}t_{4111} & t_{4112} & t_{4113} \\t_{4121} & t_{4122} & t_{4123}\end{bmatrix} & \begin{bmatrix}t_{4211} & t_{4212} & t_{4213} \\t_{4221} & t_{4222} & t_{4223}\end{bmatrix} & \begin{bmatrix}t_{4311} & t_{4312} & t_{4313} \\t_{4321} & t_{4322} & t_{4323}\end{bmatrix} & \begin{bmatrix}t_{4411} & t_{4412} & t_{4413} \\t_{4421} & t_{4422} & t_{4423}\end{bmatrix} \\\begin{bmatrix}t_{5111} & t_{5112} & t_{5113} \\t_{5121} & t_{5122} & t_{5123}\end{bmatrix} & \begin{bmatrix}t_{5211} & t_{5212} & t_{5213} \\t_{5221} & t_{5222} & t_{5223}\end{bmatrix} & \begin{bmatrix}t_{5311} & t_{5312} & t_{5313} \\t_{5321} & t_{5322} & t_{5323}\end{bmatrix} & \begin{bmatrix}t_{5411} & t_{5412} & t_{5413} \\t_{5421} & t_{5422} & t_{5423}\end{bmatrix}\end{bmatrix}} & \;\end{matrix}$Examples of Transpose Operations for 4-Tensors

Various types of “Transpose” operations involving self-invertingindex-exchange operations for one or two pairs of indices can be definedfrom (4·3·2·1)−=23 index re-organizations overall.

Perhaps some of the most useful of these would include:

-   -   “2134-Transpose”: Exchanging rows and columns within the outer        matrix structure (i.e., exchanging order of first two indices)        V_(jkmn)=t_(jkmn); t_(kjmn)=V_(jkmn);    -   “1243-Transpose”: Exchanging rows and columns within the inner        matrix structure (i.e., exchanging order of last two indices)        V_(jknm)=t_(jkmn); t_(jknm)=V_(jkmn);    -   “2143-Transpose”: Exchanging rows and columns within the outer        matrix structure (exchanging order of first two indices) and        exchanging rows and columns within the inner matrix structure        (exchanging order of last two indices) together        V_(kjnm)=t_(jkmn); t_(kjnm)=V_(jkmn);    -   “3412-Transpose”: Exchanging row-column pair of outer matrix        structure (first two indices) with the row-column pair of inner        matrix structure (last two indices) V_(mnjk)=t_(jkmn);        t_(mnjk)=V_(jkmn);    -   “1324-Transpose”: Grouping the row indices of both the inner and        outer matrix structures (first and third indices) followed by        grouping the column indices of both the inner and outer matrix        structures (second and fourth indices) V_(jmkn)=t_(jkmn);        t_(jmkn)=V_(jkmn).

Incidentally it is noted, for example that:

${\mathbb{T}}\overset{3412 - {Transpose}}{\longrightarrow}\Psi$

-   -   (“matrix analogy” to “point spread function” re-organization        from above)

$\Psi\overset{3412 - {Transpose}}{\longrightarrow}{\mathbb{T}}$

-   -   (“point spread function” to “matrix analogy” re-organization)

${\mathbb{T}}\overset{1324 - {Transpose}}{\longrightarrow}\Phi$

-   -   (“matrix analogy” to “variables-separable” re-organization from        above)

$\Phi\overset{1324 - {Transpose}}{\longrightarrow}{\mathbb{T}}$

-   -   (“variables-separable” to “matrix analogy” re-organization)        The “Identity” 4-Tensor

As with an N×N “Identity” matrix employing the mapping

$y_{j} = {\sum\limits_{n = 1}^{N}{a_{mn}x_{n}}}$to map an N-dimensional vector to a copy of itself, usingα_(mn)=δ_(mn)where δ_(pq) is the “Kronecker delta”

$\delta_{pq} = \left\{ \begin{matrix}0 & {{{if}\mspace{14mu} p} \neq q} \\1 & {{{if}\mspace{14mu} p} = q}\end{matrix} \right.$an “Identity” 4-tensor (for example J=M and K=N) mapping a M×N matrix toan M×N copy of itself results from employing the mapping:

$q_{ik} = {\sum\limits_{m = 1}^{M}{\sum\limits_{n = 1}^{N}{t_{jkmn}s_{mn}}}}$witht_(jknm)=δ_(jm)δ_(km).

Note that with this (variables-separable) structure gives q_(jk)=s_(jk)for each 1≤j≤M, 1≤k≤N.

Using the “matrix-of-matrices” representation, a (3×3)×(3×3) Identity4-tensor

_((3×3)×(3×3)) would have the form:

${\mathbb{I}}_{{({3 \times 3})} \times {({3 \times 3})}} = \begin{bmatrix}\begin{bmatrix}1 & 0 & 0 \\0 & 0 & 0 \\0 & 0 & 0\end{bmatrix} & \begin{bmatrix}0 & 1 & 0 \\0 & 0 & 0 \\0 & 0 & 0\end{bmatrix} & \begin{bmatrix}0 & 0 & 1 \\0 & 0 & 0 \\0 & 0 & 0\end{bmatrix} \\\begin{bmatrix}0 & 0 & 0 \\1 & 0 & 0 \\0 & 0 & 0\end{bmatrix} & \begin{bmatrix}0 & 0 & 0 \\0 & 1 & 0 \\0 & 0 & 0\end{bmatrix} & \begin{bmatrix}0 & 0 & 0 \\0 & 0 & 1 \\0 & 0 & 0\end{bmatrix} \\\begin{bmatrix}0 & 0 & 0 \\0 & 0 & 0 \\1 & 0 & 0\end{bmatrix} & \begin{bmatrix}0 & 0 & 0 \\0 & 0 & 0 \\0 & 1 & 0\end{bmatrix} & \begin{bmatrix}0 & 0 & 0 \\0 & 0 & 0 \\0 & 0 & 1\end{bmatrix}\end{bmatrix}$

Such a (3×3)×(3×3) Identity 4-tensor would map, a 3×3 pixel source imageS to a 3×3 pixel result image Q with Q=S.

More generally Identity 4-Tensors map an M×N matrix to an M×N matrix,but the matrices need not individually (row-column) “symmetric” —that isone does not require M=N.

For example, using the “matrix-of-matrices” representation, a(3×2)×(3×2) Identity 4-Tensor

_((3×2)×(3×2)) that maps a 3×2 matrix to a 3×2 matrix would have theform:

${\mathbb{I}}_{{({3 \times 2})} \times {({3 \times 2})}} = \begin{bmatrix}\begin{bmatrix}1 & 0 \\0 & 0 \\0 & 0\end{bmatrix} & \begin{bmatrix}0 & 1 \\0 & 0 \\0 & 0\end{bmatrix} \\\begin{bmatrix}0 & 0 \\1 & 0 \\0 & 0\end{bmatrix} & \begin{bmatrix}0 & 0 \\0 & 1 \\0 & 0\end{bmatrix} \\\begin{bmatrix}0 & 0 \\0 & 0 \\1 & 0\end{bmatrix} & \begin{bmatrix}0 & 0 \\0 & 0 \\0 & 1\end{bmatrix}\end{bmatrix}$

Such a (3×2)×(3×2) Identity 4-tensor would map a 3×2 pixel source imageS to a 3×2 pixel result image Q with Q=S.

For each of these Identity 4-tensor examples, regarding Remark 1 above(as to interpreting the values of elements in the interior matrix withblock-position index {j, k} in an (M×N)×(M×N) “matrix of matrices” asrepresenting a “heat map” of responsiveness of an observed or measuredsensor pixel q_(jk) in the observed or measured image matrix Q to sourceimage pixels s_(mn) in the image source matrix S), the structure of anM×N×M×N Identity 4-tensor is crystal clear as to it renderingq_(jk)=s_(jk) each 1≤j≤M, 1≤k≤N.

Re-Indexing and Reorganization a 4-Tensor-Operator Matrix-to-Matrix(Image-to-Image) Equation as a Matrix-Operator Vector-to-Vector Equation

Although in an image the row and column ordering, two-dimensionalneighboring arrangement of pixels, and other such two-dimensionalindexing details are essential, some linear transformations act entirelyindependently of the two-dimensional index structure. An example, aresituations where one can regard the relationships defined by a tensormapping between matrices such as

Q=

S

as simply representing a set of simultaneous equations

$q_{jk} = {\sum\limits_{m = 1}^{M}{\sum\limits_{n = 1}^{N}{t_{jkmn}s_{mn}}}}$1 ≤ j ≤ J, 1 ≤ k ≤ K.

In such circumstances one could without consequence uniquely re-indexthe variables with an indexing scheme that serializes the index sequencein an invertible way. For example, one can define two serializingindices p and q to serialize a J×K×M×N dimensional 4-tensor

comprising elements t_(jkmn) into JK×MN dimensional matrix T comprisingelements t_(pq) using the index-mapping relationsp=(j−1)K+kr=(m−1)N+nthose relations can be inverted viaj=Mod(p−1,K)+1k=Floor(p ⁻¹ /K)+1=Ceiling [p/K]m=Mod(r−1, N)+1n=Float(r ⁻¹ /N)+1=Ceiling[r /N]

Using these, one can define the serialized-index vectors q, comprisingelementsq _(p)1≤p≤JK,and s, comprising elementsS _(r)1≤r≤MN,which are simply “scanned” or “flattened ”versions of matrix Q,comprising elementsq _(jk)1≤j≤J, 1≤k≤Kand matrix S, comprising elementss _(mn)1≤m≤M, 1≤n≤N

An example “scanning”or “flattening” index correspondence isq _((j−1)K+k) ↔q _(jk)1≤j≤J, 1≤k≤≤Ks _((m−1)N+m) ↔s _(mn)1≤m≤M, 1≤n≤Nand its corresponding inverse correspondence isq _(p) ↔q _(Mod(p−1,N)+1,Ceiling(p/K)), 1≤p≤JKs _(r) ↔S _(Mod(r−1,N)+1,Ceiling(r/N)), 1≤r≤MN.

The last pair of these index correspondences can be used to formallydefine index-serializing mappingsq _(p) =q _(Mod(p−1,N)+1,Ceiling(p/K)), 1≤p≤JKs _(r) =s _(Mod(r−1,N)+1,Ceiling(r/N)), 1≤r≤MNthat provide a flattening reorganization of the elements q_(jk)comprised by the J×K-dimensional matrix Q into a vector q comprisingelements q_(p), and a flattening reorganization of the elements s_(mn)comprised the M×N-dimensional matrix S into a vector s comprisingelements s_(r).These result in flattening transformations Q→q and →s.

The first pair of the index correspondences can be used to formallydefine index-vectorizing mappingsq _(jk) =q _((j−1)K+k) 1≤j≤J, 1≤k≤Ks _(mn) =s _((m−1)N+N) 1≤m≤M, 1≤n≤Nthat provide a partitioning reorganization of the elements q_(p) ofvector q into the elements q_(jk) comprised by the J×K-dimensionalmatrix Q, and a partitioning reorganization of the elements s_(r) ofvector q into the elements s_(mn) comprised the M×N-dimensional matrixS. These result in partitioning transformations and q→Q and s→S whichreconstruct the matrices Q and S from the serialized vectors q and s.

In a corresponding way, one can use these same serialized-indices tocorrespondingly re-label and reorganize the values of the(J×K)×(M×N)-dimensional tensor

to the JK×MN-dimensional matrix T. The mapping

→T is given byt _((j−1)K+k, (m−1)N+n) =t _(jkmn)1≤j≤J, 1≤k≤K, 1≤m≤M, 1≤n≤Nand the reverse mapping T→

is given byt _(Mod(p−1,K)+1, Ceiling(p/K), Mod(r−1,K)+1, Ceiling(r/N) t _(pr)1≤p≤JK, 1≤r≤MN

Thus, because of the transformational equivalence between

$q_{jk} = {\sum\limits_{m = 1}^{M}{\sum\limits_{n = 1}^{N}{t_{jkmn}{s_{mn}\left( {{Matrix} - {{Tensor}\mspace{14mu}{equation}}} \right)}}}}$and$q_{p} = {\sum\limits_{p = 1}^{M*N}{t_{pr}{s_{r}\left( {{Vector} - {{Matrix}\mspace{14mu}{equation}}} \right)}}}$for the same (but re-indexed) variables, this allows one to exactlyrepresent the matrix-tensor equationQ=

Sas an equivalent vector-matrix equationq=Ts

More generally, the index serialization functions can be arbitrary aslong as they are one-to-one and onto over the full range and domain ofthe respective indices, and invertably map pairs of integers to singleintegers. For example they could be organized as a scan in other ways,or even follow fixed randomly-assigned mapping. In general one canwrite:

$q_{jk} = {{{\sum\limits_{m = 1}^{M}{\sum\limits_{n = 1}^{N}{t_{jkmn}s_{mn}}}}\overset{\begin{matrix} = \\ = \end{matrix}}{->}q_{p}} = {\sum\limits_{p = 1}^{M*N}{t_{pr}s_{r}}}}$ and$q_{p} = {{{\sum\limits_{p = 1}^{M*N}{t_{pr}s_{r}}}\overset{\begin{matrix} = \\ = \end{matrix}}{->}q_{jk}} = {\sum\limits_{m = 1}^{M}{\sum\limits_{n = 1}^{N}{t_{jkmn}s_{mn}}}}}$or more compactly

$Q = {{{\mathbb{T}}\;{S\overset{{index}\mspace{14mu}{serialization}}{\longrightarrow}q}} = {Ts}}$$q = {{{Ts}\overset{{index}\mspace{14mu}{vectorization}}{\longrightarrow}Q} = {{\mathbb{T}}\; S}}$or more abstractly

$Q = {{{\mathbb{T}}\;{S\overset{{Array}\mspace{14mu}{Flatten}}{\longrightarrow}q}} = {Ts}}$$q = {{{Ts}\overset{{Array}\mspace{14mu}{Partition}}{\longrightarrow}Q} = {{\mathbb{T}}\;{S.}}}$

This is extremely valuable as it allows for matrix met solve inverseproblems or implement transformations on images in terms of matrices. Ofcourse matrix methods have been used in variables-separable imageprocessing for decades employing various ad hd constructions. Those adhoc constructions could be formalized with the aforedescribed 4-tensorrepresentation should one be interested in the exercises, but moreimportantly the computation of the aforedescribed 4-tensorrepresentation and the formal isomorphic equivalence between 4-tensorlinear transformations mapping matrices (representing images) tomatrices (representing images) and matrix transformations mappingvectors to vectors allows clarity and methodology to complicatednon-variables-separable linear imaging transformation, inverses,pseudo-inverses. etc. Also importantly the aforedescribed 4-tensorrepresentation readily extends to mappings among tensors as may beuseful in color, multiple-wavelength, tomographic, spatial data, and anyother settings and applications.

Additionally, as an aside: the aforedescribed 4-tensor representationnaturally defines eigenvalue/eigenmatrix and eigenvalue/eigentensorproblems; for example the eigenvalue/eigenmatrix problem

Z _(i) =λ _(i) Z _(i)1≤i≤JKfor

a J×K×J×K 4-tensor, the collection of indexed scalars {λ₁} 1≤i≤JK thescalar eigenvalues, and the collection of indexed matrices {Z_(i)}1≤i≤JK the eigenmatrices is equivalent to the eigenvalue/eigenvectorproblemTz _(i)≤λ_(i) z _(i) 1≤i≤JKvia

${{\mathbb{T}}\overset{{Array}\mspace{14mu}{Flatten}}{\longrightarrow}T};{Z_{i}\overset{{Array}\mspace{14mu}{Flatten}}{\longrightarrow}z_{i}}$for calculation and analysis and transformed back via

${T\overset{{Array}\mspace{14mu}{Partition}}{\longrightarrow}{\mathbb{T}}};{z_{i}\overset{{Array}\mspace{14mu}{Partition}}{\longrightarrow}Z_{i}}$

These general process can be order-extended and further generalized tosimilarly transform eigenvalue/eigentensor problems into equivalenteigenvalue/eigenvector problems, and extended further in various ways toreplace the eigenvalue scalars with an “eigenvalue array.”

As and additional aside, these same and similar approaches employing

${T\overset{{Array}\mspace{14mu}{Partition}}{\longrightarrow}{\mathbb{T}}};{z_{i}\overset{{Array}\mspace{14mu}{Partition}}{\longrightarrow}Z_{i}}$${{\mathbb{T}}\overset{{Array}\mspace{14mu}{Flatten}}{\longrightarrow}T};{Z_{i}\overset{{Array}\mspace{14mu}{Flatten}}{\longrightarrow}z_{i}}$and other combined or more generalized reorganization methods

${\mathbb{T}}\overset{{{Tensor}\mspace{14mu}{Index}\mspace{14mu}{Serialization}},{{Index}\mspace{14mu}{Vectorization}},{{Index}\mspace{14mu}{Reoranization}}}{\longrightarrow}{\mathbb{V}}$can be order-extended and further generalized to similarly transform thevast understanding, rules, bases, transformations, vector spaces, spacesof matrices, properties of matrices, and matrix-vector equations into awide range of tensor understandings, tensor rules, tensor bases, tensortransformations, and properties of tensors, spaces of tensors, andtensor-matrix and tensor-tensor equations.

Attention is next directed to inversion and then to image formation, andthen after first developing and using extensions of the aforedescribed4-tensor representation to mappings among tensors) expanding these tocolor/multiple-wavelength imaging applications.

Inverse of a 4-Tensor

Accordingly, forQ=

S with M=J, N=K,if all the represented individual equations are linearly independent andof full rank, then the matrix T defined by

${{\mathbb{T}}\overset{{Array}\mspace{14mu}{Flatten}}{\longrightarrow}T};{T\overset{{Array}\mspace{14mu}{Partition}}{\longrightarrow}{\mathbb{T}}}$is invertible and the pixel values of the source image S can, beobtained from the pixel values of the measurement Q by simply invertingthe matrix T:s=T ⁻¹ qwhere the corresponding “flattening” and “partitioning” indextransformations are employed among the matrices and vectors

${S\overset{{Array}\mspace{14mu}{Flatten}}{\longrightarrow}s};{s\overset{{Array}\mspace{14mu}{Partition}}{\longrightarrow}S}$${Q\overset{{Array}\mspace{14mu}{Flatten}}{\longrightarrow}q};{{q\overset{{Array}\mspace{14mu}{Partition}}{\longrightarrow}Q}.}$

Further, the pixel values of the source image S can be obtained from thepixel values of the measurement Q by simply inverting the matrix T toobtain T⁻¹, multiplying the flattened measurement data q with T⁻¹ toobtain the vector s, and partitioning the result into the source (image)matrix S:

$Q\overset{{Array}\mspace{14mu}{Flatten}}{\longrightarrow}q\overset{T^{- 1}q}{\longrightarrow}s\overset{{Array}\mspace{14mu}{Partition}}{\longrightarrow}S$

It is noted that effectively the column vectors of the matrix T serve asthe natural linearly-independent spanning basis of the composite sensorand optical arrangement corresponding to a particular positioningsituation. The natural linearly-independent spanning basis is notnecessarily orthogonal, although it can of course be orthogonalized ifuseful using Gram-Schmitt of other methods. Additionally, the naturallinearly-independent spanning basis can be transformed into othercoordinate systems defined by other basis functions should that beuseful. Such transformations can include the effects of discrete Fouriertransforms, wavelet transforms, Walsh/Hadamard transforms, geometricrotations and scaling transforms, etc.

The simple approach employing T⁻¹ reconstructs the image by simplyreproducing individual columns of an identity matrix, more precisely adiagonal matrix whose non-zero diagonal elements represent the lightamplitude at a particular pixel. The invention provides for thereplacement of this simple approach with other methods fitting into thesame structure or delivering the same effect; for example projectiontechniques matched filters, generalized inverses, SVD operations, sparsematrix operations, etc. These can be formatted in Tensor or matrixparadigms in view of the formal transformational tensor/matrixisomorphism established above. An example of this, namely the pseudoinverse case of a generalized inverse operation.

It is noted that the matrix T can become quite large, making inversionand subsequent operations described above numerically andcomputationally challenging. The invention provides for separatingmatrix T operations into smaller blocks (for example JEPG and MPEGregularly employ 8×8 and 16×16 blocks). The invention provides for theseblocks to be non-overlapping, to overlap, and to be interleaved. Theinvention further provides for blocked inversion results involvingoverlapping blocks or interleaving blocks to be combined by linear orother operations to suppress block-boundary artifacts.

Pseudo-Inverse of a 4-Tensor

Further, because in image capture a system usually spatially quantizesnatural source image without a pixel structure, it is additionalpossible to measure a larger number of pixels than will be used in thefinal delivered image, that is M<J and N<K.

In traditional image processing such an excess-measurement scheme can beused in various “oversampling” methods, or could be decimated viaresampling. Instead of these, the excess measurements can be used tocreate an over-specified system of equations that provides otheropportunities. For example the resulting over-specified matrix T can beused to generate a generalized inverse T⁺.

For example, if the 4-tensor

represents a transformation of a 2-dimensional (monochromatic) “sourceimage” represented as an M×N matrix of “brightness” values:

$S = \begin{bmatrix}s_{11} & \ldots & s_{1N} \\\vdots & \; & \vdots \\s_{M\; 1} & \ldots & s_{MN}\end{bmatrix}$to a J×K array of measured 2-dimensional (monochromatic) image datarepresented as a J×K matrix of “brightness” values:

$Q = \begin{bmatrix}q_{11} & \ldots & q_{1K} \\\vdots & \; & \vdots \\q_{J\; 1} & \ldots & q_{JK}\end{bmatrix}$with M<J, N≤K, via

$q_{jk} = {\sum\limits_{m = 1}^{M}{\sum\limits_{n = 1}^{N}{t_{jkmn}s_{mn}}}}$1 ≤ j ≤ J, 1 ≤ k ≤ K.represented asQ=TSthen a pseudo-inverse tensor

can be defined via:

${\mathbb{T}}\overset{{Array}\mspace{14mu}{Flatten}}{\longrightarrow}T\overset{{Psuedo}\text{-}{Inverse}\mspace{14mu}{Formulaiton}}{\longrightarrow}T^{+}\overset{{Array}\mspace{14mu}{Partition}}{\longrightarrow}{\mathbb{T}}^{+}$and represented asS=T⁺Q

Further, the pixel values of the source image S can be obtained from thepixel values of the measurement Q by forming the pseudo-inverse of thematrix T, multiplying the flattened measurement data q with T⁺ to obtainthe vector s, and partitioning the result into the source (image) matrixS:

$Q\overset{{Array}\mspace{14mu}{Flatten}}{\longrightarrow}q\overset{T^{+}q}{\longrightarrow}s\overset{{Array}\mspace{14mu}{Partition}}{\longrightarrow}S$

There are a number of pseudo-inverses and related singular-valuedecomposition operators, but of these it can be advantageous for theoptical imaging methods to be described for the generalized inverse T⁺to be specifically the “Moore-Penrose” generalized (left) inversedefined (when a matrix T has all linearly-independent columns) using thematrix transpose T^(T) or conjugate transpose T^(†) of T and matrixinverse operations as:T⁺=(T^(T)T)⁻¹T^(T) for real-valued T

T³⁰ =(T^(†)T)⁻¹T⁵⁵⁴ for complex-valued T

(There is also Moore-Penrose generalized “right” inverse defined when amatrix T has all linearly-independent rows.) The Moore-Penrosegeneralized inverse inherently provides a “Least-Squares” statisticalfit where solvable subsets of the larger number of equations givedifferent inconsistent solutions. This “Least-Squares” statistical fitcan provide robustness to the imaging system, for example in the casewhere one or more sensor elements degrade, are damaged, are occulted bydirt, are occulted by objects, are altered by transparent or translucentdroplets or deposits, etc.

Using the Moore-Penrose generalized inverse for real-valued pixelquantities, the pixel values of the source image S can be obtained fromthe pixel values of the measurement Q by forming the pseudo-inverse ofthe matrix T, multiplying the flattened measurement data q with T⁺ toobtain the vector s, and partitioning the result into the source (image)matrix S:

$Q\overset{{Array}\mspace{14mu}{Flatten}}{\longrightarrow}q\overset{{({T^{\tau}T})}^{- 1}r^{r}q}{\longrightarrow}s\overset{{Array}\mspace{14mu}{Partition}}{\longrightarrow}S$Configurations for Applications

Drawing on the functionality described above and taught in theInventor's related lensless imaging patent filings listed at thebeginning of this application, a wide range of additional provisions andconfigurations can be provided so as to support of vast number ofvaluable and perhaps slightly revolutionary imaging applications.

In an example generalizing assessment, the invention provides for arigid or flexible surface to be configured to implement a lenslesslight-field sensor, producing electrical signals that can be used inreal time, or stored and later retrieved, and provided to acomputational inverse model algorithm executing on computationalhardware comprising one or more computing elements so as to implement alensless, light-field camera.

In another aspect of the invention, a rigid surface is configured toadditionally function as a housing and thus operate as a “seeinghousing”.

In another aspect of the invention, a rigid surface is configured toadditionally function as a protective plate and thus operate as a“seeing armor”.

In another aspect of the invention, a rigid surface is configured toadditionally function as an attachable tile and thus operate as a“seeing tile”.

In another aspect of the invention, a rigid surface is configured toadditionally function as an attachable film and thus operate as a“seeing film”.

In another aspect of the invention, a flexible surface is configured toadditionally function as an attachable film and thus operate as a“seeing film”.

In another aspect of the invention, a flexible surface is configured toadditionally function as a garment and thus operate as a “seeinggarment”.

In another aspect of the invention, a flexible surface is configured toadditionally function as a shroud and thus operate as a “seeing shroud”.

In another aspect of the invention, a flexible surface is configured toadditionally function as an enveloping skin and thus operate as a“seeing skin”.

In another aspect, of the invention, the rigid or flexible surface issmall in size.

In another aspect of the invention, the rigid or flexible surface islarge in size.

In another aspect of the invention, the rigid or flexible surface isflat.

In another aspect of the invention, the rigid or flexible surface iscurved.

In another aspect of the invention, the rigid or flexible surface isrendered as a polytope.

In another aspect of the invention, the rigid or flexible surface isrendered as a dome.

In another aspect of the invention, the rigid or flexible surface isrendered as a part of a sphere.

In another aspect of the invention, the rigid or flexible surface isrendered as a part of a spheroid.

In another aspect of the invention, the rigid or flexible surface isrendered as a sphere.

In another aspect of the invention, the rigid or flexible surface isrendered as a spheroid.

In another aspect of the invention, the rigid or flexible surface istransparent.

In another aspect of the invention, the rigid or flexible surface istranslucent.

In another aspect of the invention, the rigid or flexible surface isopaque.

In another aspect of the invention, the rigid or flexible surfaceperforms contact sensing.

In another aspect of the invention, the rigid or flexible surface isconfigured to perform contact sensing with near-zero separationdistance.

In another aspect of the invention, the rigid or flexible surface isconfigured to perform contact image sensing with zero separationdistance.

In another aspect of the invention, the rigid or flexible surfaceperforms distributed optical imaging.

In another aspect of the invention, the rigid or flexible surfaceperforms distributed optical sensing.

In another aspect of the invention, the rigid or flexible surfaceperforms image sensing of ultraviolet light.

In another aspect of the invention, the rigid or flexible surfaceperforms image sensing of infrared light.

In another aspect of the invention, the rigid or flexible surfaceperforms image sensing of selected ranges of visible color light.

In another aspect of the invention, the rigid or flexible surfaceperforms imaging.

In another aspect of the invention, the rigid or flexible surfaceperforms distributed chemical sensing employing optical chemical sensingproperties of at least one material.

In another aspect of the invention, the rigid or flexible surfaceperforms distributed radiation sensing employing optical radiationsensing properties of at least one material.

In another aspect of the invention, the rigid or flexible surfaceperforms distributed magnetic field sensing employing optical magneticfield sensing properties of at least one material.

In another aspect of the invention, the rigid or flexible surface isconfigured to emit light.

In another aspect of the invention, the rigid or flexible surface isconfigured to operate as a light-emitting display.

In another aspect of the invention, the rigid or flexible surface isconfigured to operate as a selectively self-illuminating contact imagingsensor.

In another aspect of the invention, the computational inverse modelalgorithm is configured to provide variable focusing.

In another aspect of the invention, the computational inverse modelalgorithm is configured to mixed depth-of-field focusing.

In another aspect of the invention, the computational inverse modelalgorithm is configured to implement a viewpoint with a controllablelocation.

In another aspect of the invention, the computational inverse modelalgorithm is configured to implement a plurality of viewpoints, eachviewpoint having a separately controllable location.

In another aspect of the invention, the computational inverse modelalgorithm is configured to provide pairs of outputs so as to function asa stereoscopic camera.

In another aspect of the invention, the computational inverse modelalgorithm is configured to capture a panoramic view.

In another aspect of the invention, the computational inverse modelalgorithm is configured to capture a 360-degree view.

In another aspect of the invention, the computational inverse modelalgorithm configured to capture a partial spherical view.

In another aspect of the invention, the computational inverse modelalgorithm is configured to capture a full spherical view.

In another aspect of the invention, the rigid or flexible surface isconfigured to perform enveloping image sensing with near-zero separationdistance.

In another aspect of the invention, the rigid or flexible surface isconfigured to perform contact enveloping sensing with zero separationdistance.

In another aspect of the invention, the rigid or flexible surface isconfigured to operate as a selectively self-illuminating envelopingimaging sensor.

In another aspect of the invention, the computational inverse modelalgorithm is configured to operate at slow-frame video rates.

In another aspect of the invention, the computational inverse modelalgorithm is configured to operate at conventional video rates.

In another aspect of the invention, the computational inverse modelalgorithm and computational hardware is configured to operate athigh-speed video rates.

Closing

The terms “certain embodiments”, “an embodiment”, “embodiment ”,“embodiments”, “the embodiment”, “the embodiments”, “one or moreembodiments”, “some embodiments”, and “one embodiment” mean one or more(but not all) embodiments unless expressly specified otherwise. Theterms “including”, “comprising”, “having” and variations thereof mean“including but not limited to”, unless expressly specified otherwise.The enumerated listing of items does not imply that any or all of theitems are mutually exclusive, unless expressly specified otherwise. Theterms “a”, “an” and “the” mean “one or more”, unless expressly specifiedotherwise.

The foregoing description, for purpose of explanation, has beendescribed with reference to specific embodiments. However, theillustrative discussions above are not intended to be exhaustive or tolimit the invention to the precise forms disclosed. Many modificationsand variations are possible in view of the above teachings. Theembodiments were chosen and described in order to best explain theprinciples of the invention and its practical applications, to therebyenable others skilled in the art to best utilize the invention andvarious embodiments with various modifications as are suited to theparticular use contemplated.

While the invention has been described in detail with reference to usedembodiments, various modifications within the scope of the inventionwill be apparent to those of ordinary skill in this technological field.It is to be appreciated that features described with respect to oneembodiment typically can be applied to other embodiments.

The invention can be embodied in other specific forms without departingfrom the spirit or essential characteristics thereof. The presentembodiments are therefore to be, considered in all respects asillustrative and not restrictive, the scope of the invention beingindicated by the appended claims rather than by the foregoingdescription and all changes which come within the meaning and range ofequivalency of the claims are therefore intended to be embraced therein.

Although exemplary embodiments have been provided in detail, variouschanges, substitutions and alternations could be made thereto withoutdeparting from spirit and scope of the disclosed subject matter asdefined by the appended claims. Variations described for the embodimentsmay be realized in any combination desirable for each particularapplication. Thus particular limitations and embodiment enhancementsdescribed herein, which may have particular advantages to a particularapplication, need not be used for all applications. Also, not alllimitations need be implemented in methods, systems, and apparatusesincluding one or more concepts described with relation to the providedembodiments. Therefore, the invention properly to be construed withreference to the claims.

CITED REFERENCES

Cited Books

-   [B1] D. J. Brady, Optical Imaging and Spectroscopy, Wiley, 2009,    ISBN 978-0-470-04823-8.-   [B2] J. R. Janesick, Photon Transfer DN→λ, SPIE Press, 2007 ISBN    978-0-819-6722-5.-   [B3] A. J. Devaney, Mathematical Foundations of Imaging, Tomography    and Wavefield Inversion, Cambridge University Press, 2012, ISBN    978-0-521-11974-0.-   [B4] C. Zhang and T. Chen, Light Field Sampling, Morgan and Claypool    2006, ISBN 978-1-598-29076-9.-   [B5] O. Scherzer, M. Grasmair, H. Grossauer, M. Haltmeier, F.    Lenten, Variational Methods in Imaging, Springer, 2009, ISBN    978-0-387-30931-6.-   [B6] M. Elad, Sparse and Redundant Representations, Springer 2010    ISBN 978-1-4419-7010-4.-   [B7] J. P. Dakin, R. G. W. Brown, eds. Handbook of Optoelectronics    Vol 1, Taylor and Francis, 2012, ISBN 978-0-7503-0646-1.-   [B8] H. Caulfied, “Holography Shadow Casting,” in E. Camatini (ed.),    Progress in Electro-Optics: Reviews of Recent Developments, NATO    Advanced Study Institute, Series B (Physics), 1973/1975.-   [B9] H. Yanai, Takeuchi, Y. Takane, Projection Matrices, Generalized    Inverse Matrices, and Singular Value Decomposition, Springer, 2011,    ISBN 978-1-4419-9886-6.-   [B10] A. Bjerhammar, Theory of Errors and Generalized Matrix    Inverses, Elsevier, 1973.-   [B11] R. R. Roa, S. K. Mitra, Generalized inverses of Matrices and    its Applications, Wiley, 1971, ISBN 0-471-70821-6.-   [B12] C. W. Groetsch, Generalized Inverses of Linear    Operators—Representation and Approximation, Marcel Dekker, Inc.,    1977, ISBN 0824766156.-   [B13] R. Piziak, P. L. Odell, Matrix Theory—From Generalized    Inverses to Jordan Form, Chapman & Hall/CRC, 2007, ISBN    978-1-58488-625-0.-   [B14] E. Chang, “The Generalized Inverse and Interpolation Theory,”    in Recent Applications of Generalized Inverses, S. L Campbell (ed.),    Pitman, 1982, ISBN 0-273-08650-6.-   [B15] A. Albert, Regression and the Moore-Penrose Pseudoinverse,    Academic Press, 1972, Library of Congress No. 72-77337 (subsequently    ISBN 0-120-48450-1).-   [B16] J. Wagner D. Keszler, R. Presley, Transparent Electronics,    Springer 2008, ISBN 978-0-378-72341-9.-   [B17] A. Facchetti, T. Marks, eds., Transparent Electronics—From    Synthesis to Applications, 2010, ISBN 978-0-470-99077-3.-   [B18] R. Chaji, A. Nathan, Thin Film Transistor Circuits and    Systems, Cambridge, 2013, ISBN 978-1-107-01233-2.-   [B19] T. Tsujimura OLED Displays—Fundamentals and Applications,    Wiley, 2012, ISBN 978-1-118-14051-2.-   [B20] G. Held, Introduction to Light Emitting Diode Technology and    Applications, CRC, 2009, ISBN 978-1-4200-7662-2.-   [B21] R. Shiner, J. Shine (eds.), Organic Electronics in Sensors and    Biotechnology, McGraw Hill, 2009, ISBN 978-0-07-159675-6, Chapter 6    and Section 5.5.-   [B22] D. R. Gamota, P. Brazis, K. Kalyanasundaram, J. Zhang, Printed    Organic and Molecular Electronics, Kluwer, 2004, ISBN 1-4020-7707-6.-   [B23] E. Cantatore (ed.), Applications of Organic and Printed    Electronics—A Technology-Enabled Revolution, Springer, 2013, ISBN    978-1-4614-3160-2.-   [B24] A. E. Javier, Solution-Processable Materials for Printable    Electronics, UMI/Proquest, 2010, ISBN: 978-1-1240-0854-7.-   [B25] Z. Cui, Printed Electronics: Materials, Technologies and    Applications, Wiley, 2016, ISBN 978-1-1189-2092-3.-   [B26] A. Facchetti, “Materials and Process Engineering for Printed    and Flexible Optoelectronic Devices,” in Frontiers of Engineering:    National Academy of Engineering, Reports on Leading-Edge Engineering    from the 2013 Symposium National Academies Press, 2014, ISBN    978-0-309-29603-8, pp 113-125.-   [B 27] B. Gnade, N. Fruehauf, B. Chalamala, J. Jang, Jin (eds),    Flexible Electronics-Materials and Device Technology, (Proceedings    769), Materials Research Society, 2003, ISBN 1558997067.-   [B28] W. Wong, Flexible Electronics: Materials and Applications,    Springer, 2009, ISBN 978-0-3877-4362-2.-   [B29] S. Logothetidis, Handbook of Flexible Organic Electronics,    Woodhead Publishing, 2014, ISBN 978-1-7824-2035-4.-   [B30] M. Caironi, Large Area and Flexible Electronics, Wiley-VCH,    2015, ISBN 978-3-5273-3639-5.-   [B31] G. Shen, Flexible Electronics: From Materials to Devices,    World Scientific Publishing Co, 2015, ISBN 978-9-8146-5198-1.-   [B32] F. Gardiner, Polymer Electronics—A Flexible Technology,    Smithers Rapra Technology, 2009, ISBN 978-1-8473-5421-1.    Cited Articles, Presentations, and Technical Papers-   [P1] A. Bjerhammar, “A Generalized Matrix Algebra” N.R.C. Can. Div.    Appl. Phys., Ottawa., 1957.-   [P2] R. D. Jansen-van Vuuren, A. Armin, A. K. Pandey, P. L. Burn, P.    Meredith, “Organic Photodiodes: The Future of Full Color Detection    and Image Sensing,” Advanced Materials, Vol 28, Issue 24, Jun. 22,    2016, pp. 4766-4802.-   [P3] H. Wiman, “Improvement of Digital Image Resolution by    Oversampling,” XVIIth ISPRS Congress—Technical Commission II:    Systems for Data Processing and Analysis, Aug. 2-14, 1992,    Washington, D.C., USA, L W. Fritz, J. R. Lucus (eds), in ISPRS    Archives—Volume XXIX Part B2, 1992, International Society for    Photogrammetry and Remote Sensing, 1993, pp. 323-327.-   [P4] D. G. Stork, P. R. Gill. “Lensless Ultra-Miniature CMOS    Computational Imagers and Sensors.” Proc. SENSORCOMM (2013):    186-190.-   [P5] M. S. Asif, et al, “Flatcam: Thin, Lensless Cameras Using Coded    Aperture and Computation.” IEEE Transactions on Computational    Imaging (2017).-   [P6] V. Boominathan, et al., “Lensless Imaging: A Computational    Renaissance,” IEEE Signal Processing Magazine 33(5), September 2016,    pp. 23-35. Available at    http://www.ece.rice.edu/˜vb10/documents/2016/Lensless_Imaging_Computation    al_Renaissance.pdf-   [P7] G. Williams, C. Backhouse, H. Aziz, “Integration of Organic    Light Emitting Diodes and Organic Photodetectors for    Lab-on-a-Chip-Detection Systems”, Electronics, Vol 3, Feb. 13, 2014,    pp. 43-75.-   [P8] F. Krujatz, O. R. Hild, K. Fehse, et al. “Exploiting the    Potential of OLED-Based Photo-Organic Sensors for Biotechnological    Applications” Chemical Sciences Journal, Vol 7, Issue 3, Jul. 18,    2016.-   [P9] M. Punke, S. Mozer, M. Stroisch, “Organic Semiconductor Devices    for Micro-Optical Applications”, Proc of Spie, Vol 6185, Apr. 3,    2006.-   [P10] R. Bhattacharya, et al. “Organic LED Pixel Array on a Dome”,    Proceedings of the IEEE, Vol 93, Issue 7, Jul. 5, 2005, pp.    1273-1280.-   [P11] D. L. Cade, “Hitachi's Lensless Camera Uses Moire and Math,    Not Glass, to Take Photos”, Hitachi Press Release, Nov. 16, 2016.    Available at    https://petapixel.com/2016/11/16/hitachis-lensless-camera-uses-moire-math-not-glass-take-photos/    as retrieved Jun. 20, 2017.-   [P12] Hitachi, “Lensless-Camera Technology for Easily Adjusting    Focus on Video Images after Image Capture”, Hitachi Press Release,    Nov. 15, 2016. Retrieved Jun. 20, 2017 from    http://www.hitachi.com/New/cnews/month/2016/11/161115.html-   [P13] Hitachi, “Technical Explanation of “Hitachi Lensless Camera””,    Hitachi, Dec. 5, 2016 Retrieved Jun. 20, 2017 from    https://physics.stackexchange.com/questions/1296640/technical-explanation-of-hitachi-lensless-camera.-   [P14] Hitachi, “Lensless Camera Technology for Adjusting Video Focus    After Image Capture”, Hitachi Press Release, Nov. 21, 2016,    Retrieved Jun. 20, 2017 from    https://phys.org/news/2016-11-lensless-camera-technology-adjusting-video.html-   [P15] Sumito, “Organic Photodiodes”, Sumito Chemical Printed    Electronics, visited on Jun. 20, 2017. Retrieved Jun. 20, 2017 from    https://www.sumitomo-chem.co.jp/printedelectronics/en/application/photodiodes.html.-   [P16] Fraunhofer Research Institution for Organics, Materials And    Electronic Devices, “Smart Optoelectronic Micro-Sensors By    OLED-On-Silicon”, Fraunhofer Comedd. Available at    https://www.comedd.frauhofer.de/content/dam/comedd/common/products/COM    EDD/oledcmos-e.pdf retrieved Jun. 20, 2017.)-   [P17] M. G. Han, K. B. Park, et al. “Narrow-Band Organic Photodiodes    for High-Resolution Imaging”, Applied Materials & Interfaces, Vol 8,    Issue 39, Sep. 13, 2016, pp. 26143-26151.-   [P18] W. Wang, F. Zhang, et al. “Highly Narrowband    Photomultiplication Type Organic Photodetectors”, Nano Letters, Vol    17, Issue 3, pp. 1995-2002, Feb. 6, 2017.-   [P19] D. H. Kim, K. S. Kim, et al. “A High Performance    Semitransparent Organic Photodetector with Green Color Selectivity”    Applied Physics Letter, Vol 105, 2014.-   [P20] R. Hany, “Transparent Organic Photodiodes”, EMPA, Retrieved    Jun. 20, 2017 from https://www.empa.ch/web/s209/organic-photodiodes.-   [P21] F. Arca, Organic Photodiodes for Industrial Sensing and    Medical Imaging, Dissertation, Technische Universität München    Lehrstuhl für Nanoelektronik, Mar. 3, 2013, Retrieved Jun. 20, 2017    from https://mediatum.ub.tum.de/doc/1197763/458492.pdf-   [P22] “Organic Photodiodes for Sensor Applications”, Fraunhofer    Comedd. May 14, 2014. Retrieved Jun. 20, 2017 from    https://phys.org/news/2014-05-photodiodes-sensor-applications.html.-   [P23] B. A. Katchman, J. T. Smith, et at “Application of Flat Panel    OLED Display Technology for the Point-of-Care Detection of    Circulating Cancer Biomarkers”, Scientific Reports, Vol 6, Article    number: 29057, Jul. 4, 2016.-   [P24] Y. Y. Noh, D. Y. Kim, “Organic Phototransistor Based on    Pentacene as an Efficient Red Light Sensor” Solid-State Electronics,    Vol 51. Issue 7, July 2007, pp. 1052-1055.-   [P25] X. Liu, E. K. Lee, et at “Flexible Organic Phototransistor    Array with Enhanced Responsivity via Metal-Ligand Charge Transfer”,    “Applied Materials & Interfaces” Vol 8, Issue 11, March 1 2016, pp.    7291-7299.-   [P26] H. W. Zan, S. C. Kao, S. R. Ouyang, “Pentacene-Based Organic    Phototransistor with High Sensitivity to Weak Light and Wide Dynamic    Range” IEEE, Vol 31, Issue 2, Jan. 19, 2010, pp. 135-137.-   [P27] K. J. Baeg, M. Binda, “Organic Light Detectors: Photodiodes    and Phototransistors”, Advanced Materials, Volume 25, Issue 31, Aug.    21, 2013, Pages 4267-4295.-   [P28] A. Koppelhuber, O. Bimber, “Towards a Transparent, Flexible,    Scalable and Disposable Image Sensor Using Thin Film Luminescent    Concentrators” Optics Express, Vol 21, Issue 4, 2013, pp. 4796-4810.-   [P29] A. Pierre, A. Gaikwad, A. C. Arias, “Charge-integrating    organic heterojunction phototransistors for wide-dynamic-range image    sensors,” Nature Photonics 11, 2017, pp.193-199.-   [P30] M. Ramuz, L. Bürgi, P. Seitz, “High sensitivity organic    photodiodes with low dark currents and increased lifetimes,” Organic    Electronics, Vol. 9, Issue 3, June 2008, pp. 369-376.-   [P31] A. Busboom, H. Elders-Boll, H. Schotten, “Uniformly Redundant    Arrays,” Experimental Astronomy, June 1998, Volume 8, Issue 2,    pp.97-123.-   [P32] R. Marcia, Z. Harmany R. Willett, “Compressive Coded Aperture    Imaging,” Computational Imaging VII, SPIE Proceedings Vol. 7246    (72460G), Feb. 3, 2009.-   [P33] W. H. Richardson. “Design of an Ultrasensitive Active Pixel    Sensor That is Based on Silicon Nanostructures,” SPIE DSS    Conference, Micro- and Nanotechnology Sensors, Systems, and    Applications III, Paper 8031-92, (May 2011).-   [P34] M. Hirsch, BiDi Screen: Depth and Lighting Aware Interaction    and Display MS Thesis, MIT, Aug. 13, 2009 (Available at    https://dam-prod.media.mit.edu/x/files/thesis/2009/mhirsch-ms.pdf as    retrieved Jul. 2, 2017.)-   [P35] M. Grosse, G. Wetzstein, A. Grundhoefer, O. Bimber, “Coded    Aperture Projection,” ACM Transactions on Graphics 29(3). Volume 29    Issue 2, March 2010; also ACM SIGGRAPH, Jul. 25-29, 2010. (Available    at http://web.media.mit.edu/˜gordonw/CodedApertureProjection/ as    retrieved Jul. 2, 2017.)-   [P36] T. Barribeau, “Shooting Full Panoramas Is Easy with Bendable    Flexcam Camera,” Imaging Resource, Aug. 19, 2013, (Available at    http://www.imaging-resource.com/news/2013/08/19/Shooting-full-panoramas-is-easy-with-bendable-Flexcam-camera    as retrieved Jul. 1, 2017.)-   [P 37] H. Everts, “A Flexible Camera: A Radically Different Approach    to Imaging” Columbia Engineering, Apr. 13, 2016 (Available at    http://engineering.columbia.edu/flexible-camera-radically-different-approach-imaging    as retrieved Jul. 1, 2017)-   [P 38] D. Sims, Y. Yue, S. Nayar, “Towards Flexible Sheet Cameras:    Deformable Lens Arrays with Intrinsic Optical Adaptation,” IEEE    International Conference on Computational Photography (ICCP),    May 2016. (Available at    http://.cs.columbia.edu/CAVE/projects/flexible_sheet_cameras/Sims_ICCP1    6.pdf as retrieved Jul. 1, 2017))-   [P39] R. Perkins, “Ultra-Thin Camera Creates Images Without Lenses,”    CalTech, Jun. 21, 2017. (Available at    http://www.caltech.edu/news/ultra-thin-camera-creates-images-without-lenses-78731    as retrieved Jul. 2, 2017.)-   [P40] R. Ng, M. Levoy, M. Bredif, G. Duval, M. Horowitz, P.    Hanrahan, “Light Field Photography with a Hand-Held Plenoptic    Camera,” Stanford University Computer Science Tech Report CSTR    2005-02, 2005. (Available at    https://graphics.stanford.edu/papers/lfcamera/lfcamera-150dpi.pdf    visited Jul. 2, 2017.)-   [P41] R. Butler, “Lytro Light Field Camera first look with Ren Ng,”    Digital Photography Review, Oct. 19, 2011. (Available at    https://www.dpreview.com/articles/7237351494/lytro-light-field-camera-first-look-with-ren-ng    visited Jul. 2, 2017.)-   [P42] V. Koifman, “Toshiba Announces Light Field Camera Module,”    Image Sensors World, Dec. 27, 2012. (Available    http://image-sensors-world.blogspot.com/2012/12/toshiba-announces-light-field-camera.html    as retrieved Jul. 2, 2017.)-   [P43] LightField Forum, “New Light Field Tech to use Sensor Layers    instead of Microlenses,” LightField Forum, May 15, 2016, (Available    at    http://lightfield-forum.com/2016/03/new-light-field-tech-to-use-transparent-sensor-layers-instead-of-microlenses/    as retrieved Jul. 2, 2017.) [P44] W. Wang, “Optical Detectors,”    slide presentation (date unknown). Available at    http://depts.washington.edu/mictech/optics/sensors/detector.pdf as    retrieved Jul. 2, 2017.-   [P45] E. Fenimore, “Coded Aperture Imaging: The Modulation Transfer    Function for Uniformly Redundant Arrays,” Appl Opt 19 (14), Jul. 15,    1980, pp,2465-2471.-   [P46] M. Levoy, “Light fields and computational imaging,” Computer,    vol. 39, no. 8, August 2006, pp. 46-55.-   [P47] A. Zomet and S. K. Nayar, “Lensless imaging with a    controllable aperture,” in IEEE Computer Society Conference on    Computer Vision and Pattern Recognition, vol. 1, 2006, pp. 339-346.-   [P48] M. Duarte, M. Davenport, D. Takhar, J. Laska, T. Sun, K.    Kelly, R. Baraniuk, “Single-Pixel Imaging via;Compressive Sampling,    IEEE Signal Processing Magazine, 2008, 25, pp.83-91.-   [P49] A. Nathan, A. Ahnood, M. Cole, “Flexible Electronics: The Next    Ubiquitous Platform,” Proceedings of the IEEE, Volume: 100 Issue:    Special Centennial Issue, May 13, 2012, pp. 1486-1517.-   [P50] Phys Org, “Researchers Achieve Major Breakthrough in Flexible    Electronics,” Phys Org News, Jan. 13, 2017, available at    https://phys.org/news/2017-01-major-breakthrough-flexible-electronics.html    retrieved Jul. 8, 2017.-   [P51] Phys Org, “New Hybrid Inks for Printed, Flexible Electronics    Without Sintering,” Phys Org News, Apr. 10, 2017, available at    https://phys.org/news/2017-04-hybrid-inks-flexible-electronics-sintering.html    as retrieved Jul. 8, 2017.-   [P52] Interuniversitair Micro-Electronica Centrum (IMEC), “Thin-film    flexible electronics,” Available at    https://www.imec-int.com/en/thin/film--flexible-electronics    retrieved Jul. 8, 2017.-   [P53] A. Abouraddy, O. Shapira, M. Bayindir, J. Arnold, F. Sorin, D.    Hinczewski, J. Joannopoulos, Y. Fink, “Large-scale optical-field    measurements with geometric fibre constructs,” Nature Mat. 5(7),    532-536 (2006).-   [P54] T. Ng, W. Wong, M. Chabinyc, S. Sambandan, and R. Street,    “Flexible image sensor array with bulk heterojunction organic    photodiode,” Appl. Phys. Lett. 92(21), 213303 (2008).-   [P55] T. Hayes, “Flexible image sensors move nearer to market,”    Optics.org, Jul. 16, 2013, available at    http://optics.org/news/4/7/22 as retrieved Jul. 9, 2017.-   [P56] K. Chopra, S. Major, D. Pandya, “Transparent conductors—A    Status Review,” Thin Solid Films, Volume 102, Issue 1, Apr. 8, 1983,    pp.1-46.-   [P57] F. Yu, S. Wu, X. Wang, G. Zhang, H. Luaband, L Qiu, “Flexible    and low-voltage organic phototransistors,” RSC Advances 7(19), Feb.    2017, pp, 11572-11577.-   [P58] S. Garlapati, T. Baby, S. Dehm, M. Hammed, V.    Chakravadhanula, R. Kruk, H. Hahn, S. Dasgupta, “Ink-Jet Printed    CMOS Electronics from Oxide Semiconductors,” Nano Micro Small,    Volume 11, Issue 29, Aug. 5, 2015, pp. 3591-3596.-   [P59] G. Huang, H. Jiang, K. Matthews, P. Wilford “Lensless Imaging    by Compressive Sensing,” IEEE International conference on Image    Processing, ICIP 2013, Paper #2393, (Available at    https://arxiv.org/abs/1305,7181 as retrieved Jul. 10, 2017.)-   [P60] D. Thapaa, Raahemifarb, V. Lakshminarayanan, “Less is more:    compressive sensing in optics and image science,” Journal of Modern    Optics, Vol. 62, No. 3, 2015, pp. 169-183.-   [P61] Erickson, Evan L., et al. “Miniature lensless computational    infrared imager.” Electronic Imaging 2016. 12 (2016): 1-4.-   [P62] R. H. Dicke, “Scatter-Hole Cameras for X-Rays and Gamma Rays,”    Astrophys. J. 153, L101 (1968).-   [P63] E. Fenimore and T. Cannon, “Coded Aperture Imaging with    Uniformly Redundant Arrays,” Appl. Opt. 17, 337 (1978).-   [P64] E. Fenimore, “Coded aperture imaging: predicted performance of    uniformly redundant arrays.” Applied Optics 17, 22 (1978):    3562-3570.-   [P65] S. Gottesman, “Coded Apertures: Past, Present, and Future    Application and Design.” Optical Engineering Applications.    International Society for Optics and Photonics, 2007.-   [P66] C. Zhou, S. Nayar, “Computational cameras: Convergence of    Optics and Processing.” IEEE Transactions on Image Processing 20. 12    (2011): 3322-3340.-   [P67] A. Veeraraghavan, et al. “Dappled photography: Mask enhanced    cameras for heterodyned light fields and coded aperture refocusing.”    ACM Trans. Graph. 26.3 (2007): 69.-   [P68] A. Levin, et al., “Image and depth from a conventional camera    with a coded aperture.” ACM transactions on graphics (TOG) 26.3    (2007): 70-   [P69] H. Jiang, G. Huang, P. Wilford, “Multi-view lensless    compressive imaging,” APSIPA Transactions on Signal and Information    Processing, 3, (2014) doi:10. 1017/ATSIP.2014.16-   [P70] G. Kim, et al., “Lensless Photography with only an image    sensor,” arXiv preprint arXiv:1702.06619 (2017).-   [P71] X. Yuan, et at., “Lensless compressive imaging,” arXiv    preprint arXiv:1508.03498 (2015).-   [P72] Yuan, Xin et al. Block-wise Lensless Compressive Camera,”    arXiv preprint arXiv:1701.05412 (2017).-   [P73] O. Cossairt, M. Gupta, S. Nayar, “When Does Computational    Imaging Improve Performance?” IEEE Transactions on Image Processing    22.2 (2013): 447-458.-   [P74] Pierre, Adrien, et at., “High Detectivity All-Printed Organic    Photodiodes.” Advanced Materials 27. 41 2015, pp. 6411-6417.-   [P75] Someya, Takao, et al. “Integration of Organic FETs with    Organic Photodiodes for a Large Area, Flexible, and Lightweight    Sheet Image Scanners.” IEEE Transactions on Electron Devices, 52.11    2005, pp. 2502-2511.

What is claimed is:
 1. A lensless light-field imaging system,comprising: an array of light sensing elements, each light-sensingelement comprising a light-sensing area and each light-sensing elementconfigured to generate an electrical photocurrent responsive to anamplitude of incoming light striking a light-sensing surface, eachlight-sensing surface arranged to experience angularly-varyingsensitivity responsive to a direction of each path of the incoming lightstriking the light-sensing surface; first electronics configured tointerface the array of light sensing elements with second electronics,the first electronics further configured to provide a plurality ofvoltage levels, each of the plurality of voltage level responsive to oneof the light-sensing element in the array of light sensing elements; thesecond electronics configured to convert each of the plurality ofvoltage levels into a corresponding electronically-represented digitalnumber, a result comprising a plurality of electronically-representeddigital numbers; and an algorithm configured to execute on acomputational processor, the algorithm for computing a two-dimensionalimage representation from the plurality of electronically-representeddigital numbers, the two-dimensional image representation correspondingto portion of a focused image at a separation distance value measuredperpendicular to the light-sensing surface of the one of the lightsensing elements in the array of light sensing elements, there being aplurality of separation distance values, wherein each of theelectronically-represented digital numbers are responsive to theamplitude of incoming light striking the light-sensing surface of anassociated light sensing element in the array of light sensing elementsand a plurality of focused image portions, and wherein the plurality ofseparation distance values are not a substantially same numeric value.2. The lensless light-field imaging system of claim 1, wherein the lightsensing elements of the array of light sensing elements are oriented inspace to form a curved surface.
 3. The lensless light-field imagingsystem of claim 1, wherein spatial positions of the plurality of focusedimage portions form a planar surface.
 4. The lensless light-fieldimaging system of claim 1, wherein the light sensing elements of thearray of light sensing elements are oriented in space to form a planarsurface.
 5. The lensless light-field imaging system of claim 1, whereinspatial positions of the plurality of focused image portions form acurved surface.
 6. The lensless light-field imaging system of claim 1,wherein the light sensing elements of the array of light sensingelements are oriented in space to form a first curved surface andspatial positions of the plurality of focused image portions form asecond curved surface.
 7. The lensless light-field imaging system ofclaim 1, wherein the algorithm is controlled by at least one separationdistance parameter.
 8. The lensless light-field imaging system of claim1, wherein the algorithm is controlled by a plurality of localizedseparation distance parameters.
 9. The lensless light-field imagingsystem of claim 1, wherein the first electronics comprises multiplexingelectronics.
 10. The lensless light-field imaging system of claim 1,wherein the first electronics comprises at least one transimpedanceamplifier circuit.
 11. The lensless light-field imaging system of claim1, wherein the light sensing elements comprise organic photodiodes. 12.The lensless light-field imaging system of claim 1, wherein the lightsensing elements comprise organic light emitting diodes.
 13. Thelensless light-field imaging system of claim 1, wherein the lightsensing elements comprise organic diodes that are co-optimized for bothlight emission and light sensing.
 14. The lensless light-field imagingsystem of claim 1, wherein the light sensing elements are arranged toemit light for an interval of time.
 15. The lensless light-field imagingsystem of claim 14, wherein the light sensing elements are arranged toemit light for the interval of time under control of the firstelectronics.
 16. The lensless light-field imaging system of claim 1,wherein the angularly-varying sensitivity of the light sensing elementsresults at least in part from a structure of the light sensing elements.17. The lensless light-field imaging system of claim 1, wherein theangularly-varying sensitivity of the light sensing elements results atleast in part from a structure attached to the array of light sensingelements.
 18. The lensless light-field imaging system of claim 1,wherein the array of light sensing elements are fabricated by a printingprocess.
 19. The lensless light-field imaging system of claim 17,wherein the structure attached to the array of light sensing elements isfabricated by a printing process.
 20. The lensless light-field imagingsystem of claim 17, wherein the structure attached to the array of lightsensing elements comprises segregated optical paths.
 21. The lenslesslight-field imaging system of claim 20, wherein the segregated opticalpaths are created by separating surfaces.
 22. The lensless light-fieldimaging system of claim 21, wherein the separating surfaces are at leastpartially-reflective.
 23. The lensless light-field imaging system ofclaim 21, wherein the separating surfaces are configured to facilitatesurface plasmon propagation.
 24. The lensless light-field imaging systemof claim 21, wherein at least one of the light sensing elements is colorselective.
 25. The lensless light-field imaging system of claim 23,wherein a color selective property results from a band gap property of asemiconductor device element comprised by the at least one of the lightsensing elements.
 26. The lensless light-field imaging system of claim1, wherein the algorithm comprises array multiplication of numericalvalues responsive to the plurality of electronically-represented digitalnumbers.
 27. The lensless light-field imaging system of claim 1, whereinthe algorithm comprises array multiplication of numerical valuesobtained from calculation of a generalized inverse matrix.
 28. Thelensless light-field imaging system of claim 1, wherein the algorithmcomprises array multiplication of numerical values obtained from aninterpolation.
 29. The lensless light-field imaging system of claim 1,wherein the algorithm comprises array multiplication of numerical valuesobtained from a predictive analytical model.
 30. The lenslesslight-field imaging system of claim 1, wherein the algorithm comprisesarray multiplication of numerical values derived from a predictiveanalytical model.
 31. The lensless light-field imaging system of claim1, wherein the algorithm comprises array multiplication of numericalvalues derived from empirical measurements.